Sequential Convoy
A sequential convoy processes messages for the same key in strict order while processing different keys in parallel, preserving per-entity causality without global serialization.
Definition
The sequential convoy pattern ensures ordered processing per correlation key: all messages for one order, one account, one device run sequentially through a single logical consumer, while independent keys proceed in parallel. It trades throughput proportionally to key distribution — a single hot key serializes behind itself, and the whole convoy stalls on one slow message.
Context and scope
Required in banking ledgers, inventory adjustments, and any state machine where per-entity message order is a correctness requirement. Kafka partition-by-key is the canonical transport-level implementation.
Common misconceptions
- Ordering is per topic, not per key — most brokers guarantee order per partition or queue; key-based partitioning is what creates the convoy
- Parallelism removes ordering problems — parallelism must be orthogonal to the key; any two messages sharing a key must share the lane
- Retries can be transparent — an in-place retry of a failed message blocks every later message for that key; dead-letter policies shape convoy stalls
Anti-patterns
- Load-balancing consumers round-robin and letting one entity messages race each other
- Letting a poison message block its convoy forever instead of dead-lettering and side-lining it
- Assuming timestamps in payloads enforce order — arrival order at the consumer is what processing order follows; timestamps are metadata
Related concepts
Authoritative source
https://www.enterpriseintegrationpatterns.com/patterns/messaging/SequentialConvoy.html
What it is
Some domains have causal chains: message two depends on the state message one created. A convoy is a lane per causal chain — inside the lane, first in first out; across lanes, free parallelism.
How it works
The router hashes the correlation key to a lane — partition, queue, or in-process actor. Each lane is single-threaded, so per-key order follows arrival order. Failure handling per lane: retry with the lane blocked, or skip-and-dead-letter with a recorded gap. Consumer scaling adds lanes, not threads per lane.
When to require it
Ledgers and balances, per-device command sequences, per-session state machines, and inventory where out-of-order decrements corrupt stock. Anywhere reordering is not an inconvenience but a correctness violation.
Pitfalls
Hot keys concentrate: one mega-tenant serializes its entire history through one lane. Lane-internal poison messages stall the tenant — pair with dead-letter plus reconciliation. Rebalancing partitions mid-flight reassigns lanes and can reorder messages in flight; drain before rebalance.