RMQ: the native x-delayed-message path is non-conformant — a requeue delay is discarded, and delivered messages carry a non-zero Header.Delayed

[iancooper]

Summary

The RMQ native delayed-message path (the x-delayed-message plugin exchange) is not conformant, in
two independent ways. Both are in src, both are currently tracked only as a sentence inside the #4240
ledger, and together they are why the conformance suite deliberately routes RMQ's delay through the
IAmAMessageProducer.Scheduler seam instead of the plugin.

Defect 1 — a requeue delay is discarded

src/Paramore.Brighter.MessagingGateway.RMQ.Async/RmqMessagePublisher.cs:129,
RequeueMessageAsync(Message, ChannelName, TimeSpan timeOut, …):

AddDeliveryHeaders(TimeSpan.Zero, deliveryTag, headers);   // :140 -- timeOut is never used

// To send it to the right queue use the default (empty) exchange
await _channel.BasicPublishAsync(
    string.Empty,          // :146 -- not the delay exchange
    queueName.Value,
    ...

The timeOut parameter is accepted and dropped: the delay header is hardcoded to TimeSpan.Zero, and
the republish goes to the default exchange rather than the x-delayed-message exchange that would
act on the header. So Requeue(message, delay) on RMQ redelivers immediately regardless of the delay
requested — conformance behaviour FR-2.

The RMQ.Sync gateway carries the same shape.

Defect 2 — a plugin-delivered message arrives with a non-zero Header.Delayed

A message sent through the delayed-message exchange arrives carrying Header.Delayed == delay rather
than TimeSpan.Zero. Every other transport delivers Header.Delayed == TimeSpan.Zero once the delay has
elapsed — the delay is a send-time instruction, not a property of the delivered message — so the
universal message-equivalence assertion trips on RMQ alone.

Whichever way this is settled it should be settled deliberately: either the gateway clears Delayed on
receipt, or Delayed is redefined across all transports and the other twelve gateways change.

How the conformance suite works around it

RMQ.Async / Classic, RMQ.Async / Quorum and RMQ.Sync all report FR-2 and FR-9 as conformant, but
via a wired RmqHarnessMessageScheduler, not the plugin. The provider deliberately presents a plain
(non-delay) exchange so DelaySupported == false and the gateway delegates to the scheduler seam — the
same seam proven for Kafka, Redis, MSSQL and SNS.

That is a legitimate conformance result for the seam, and it is also why these two defects produce no red
test today. The native path is simply not exercised.

Impact

• Anyone running RabbitMQ with the rabbitmq_delayed_message_exchange plugin and relying on a delayed
  requeue gets immediate redelivery. On a poison message that is a tight retry loop.
• Anyone using the plugin for delayed send gets a delivered message whose Header.Delayed disagrees
  with every other transport.
• The conformance suite cannot certify the native path at all, so neither defect can regress visibly.

Suggested direction

1. Thread timeOut through RequeueMessageAsync into AddDeliveryHeaders, and publish to the delay
   exchange when the subscription declares one (both RMQ.Async and RMQ.Sync).
2. Decide and document the Header.Delayed contract on receipt; normalise RMQ to it.
3. Add a native-path provider configuration to the conformance suite so FR-2/FR-9 are proven on the
   plugin as well as on the scheduler seam. Note the reference compose file pins a stock
   rabbitmq:*-management image; the plugin build would need reinstating for that configuration.

Related

• [Feat] Test Generator: Brighter-provided Reject/DLQ/requeue-with-delay flows as universal transport needs Generator Tests #4240 / FEAT: Generated transport conformance tests should cover new reject/dlq/requeue paths #4297 — where both defects were recorded
• RMQ gateways declare transient non-exclusive queues, which RabbitMQ 4.3 blocks by default #4355 — RMQ transient non-exclusive queues on RabbitMQ 4.3+