Competing Consumers
Competing consumers let multiple instances pull from one queue, sharing the work and scaling horizontally — with per-message atomic claim semantics preventing double processing.
Definition
The competing consumers pattern runs multiple consumer instances against a single queue: each message is delivered to exactly one, claimed atomically by the pop or lock mechanism. Throughput scales with consumer count until the queue or downstream saturates. It trades per-key ordering — any message may interleave with any other — for utilization.
Context and scope
The default scaling model of queue consumers: SQS, RabbitMQ, Kafka consumer groups. Pairs with sequential convoys when ordering per key matters.
Common misconceptions
- Competing consumers preserve arrival order — they deliberately break it; ordered domains need convloys or single consumers per key
- Adding consumers always adds throughput — scaling stops at the queue throughput ceiling or the downstream bottleneck; past it, consumers add contention
- Exactly-once delivery comes from the queue — queues offer at-least-once with redelivery; idempotent consumers make processing effectively-once
Anti-patterns
- Fan-out work to consumers that double-post side effects on redelivery without idempotency keys
- Scaling consumers past downstream database capacity and turning queue lag into lock contention
- Assuming visibility timeouts longer than worst-case processing are optional and getting double delivery
Related concepts
Authoritative source
https://www.enterpriseintegrationpatterns.com/patterns/messaging/CompetingConsumers.html
What it is
One queue, many workers. Each message has exactly one winner among the consumers pulling for it. Utilization stays high because work never waits for a specific worker.
How it works
Claim semantics differ by system: atomic pops in SQS-style queues, consumer group partitions in Kafka, prefetch acknowledgments in AMQP. A visibility or lease timeout bounds in-flight time: expired leases redeliver, so processing must be idempotent. Consumer count is the scaling dial, bounded by queue throughput and downstream capacity.
When to use it
Independent, parallelizable work: image processing, notification sends, cache invalidations — any workload without per-entity ordering requirements. When per-key ordering matters, combine the pattern with a sequential convoy rather than replacing it.
Pitfalls
At-least-once delivery makes side effects duplicate on crash-restart windows: idempotency keys or dedup windows are mandatory. Slow consumers with long leases hold messages hostage — size leases to p99 processing time. Autoscaling on queue depth overreacts to bursts; scale on drain-rate deficit instead.