Idempotent Consumer

An idempotent consumer records processed message ids so at-least-once delivery produces the same state as a single delivery, even after retries.

messaging

Definition

An idempotent consumer makes message handling safe to repeat. It records a stable message id or business key, commits that record with the business change in one transaction, and treats a duplicate delivery as already processed. The broker may deliver a message more than once; the consumer's effect remains single.

Context and scope

Idempotent consumers are required for at-least-once queues, webhooks, retries, and event-driven projections. They are the practical answer to "exactly-once delivery" claims, which usually mean at-least-once delivery plus deduplication.

Common misconceptions

Exactly-once delivery means the consumer never sees duplicates — most systems provide at-least-once delivery and expect idempotency
An in-memory Set is enough — it is lost on restart and not shared across instances
Deduplication is optional for projections — replaying an event can double counts or overwrite newer state

Anti-patterns

Writing the business row and the processed-id row in separate transactions
Keying deduplication by payload hash, which changes with harmless formatting differences
Keeping processed ids forever without a retention or partitioning strategy

Related concepts

Authoritative source

https://microservices.io/patterns/communication-style/idempotent-consumer.html

Duplicates are normal

At-least-once delivery guarantees that a message is delivered, not that it is delivered once. A consumer may crash after processing but before acknowledging, so the broker redelivers. A retry may follow a timeout even though the first attempt succeeded. An idempotent consumer turns those duplicates into one effective state change.

CREATE TABLE processed_messages (
  consumer   text NOT NULL,
  message_id text NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, message_id)
);

BEGIN;
INSERT INTO processed_messages (consumer, message_id)
VALUES ('order-projector', $1);
UPDATE order_totals SET total = total + $2 WHERE order_id = $3;
COMMIT;

The insert and the business update share a transaction. If the insert conflicts on the primary key, the message was already processed; acknowledge it without applying the update. If the transaction fails, both roll back and the broker can redeliver.

Choose a stable key

Use the broker’s message id or the producer’s domain event id, not a hash of the payload. A payload hash changes when a field is added or formatting changes, and it fails to detect semantically identical retries. The key must be stable across redeliveries and unique for the same logical event.

Retention matters. Processed ids cannot be kept forever in a high-volume system. Retain them longer than the broker’s maximum redelivery window, partition the table by time, and archive or delete old rows. If a message can be redelivered after the retention window, deduplication cannot help; the broker configuration must bound redelivery.

Make the effect idempotent too

Deduplication protects against the same message id, but a consumer may process two different messages that represent the same business action. Use idempotent operations where possible: set a value instead of incrementing blindly, or include a version or idempotency key in the domain model. For counters, store the contributing event ids or a per-aggregate version so a duplicate cannot increment twice.

When to use it

Use an idempotent consumer for any queue or webhook where redelivery is possible and the effect is not naturally idempotent. The advantage is safe retries and simpler failure recovery; the drawback is a deduplication store, transaction coupling, and retention. Compared to relying on exactly-once infrastructure, application-level deduplication is portable across brokers and makes the guarantee explicit.