Exactly-Once Semantics
Exactly-once semantics means each event affects the final result once, even after retries or failures; it requires idempotent or transactional end-to-end handling.
Definition
Exactly-once semantics is a processing guarantee that each input record influences the output exactly once, despite crashes, retries, and rebalances. It is achieved by combining at-least-once delivery with deduplication, idempotent writes, or atomic transactions that tie input offsets to output commits.
Context and scope
The phrase appears in stream processors and message brokers, but the guarantee is only as strong as the weakest link. A transactional producer writing to a non-transactional sink does not give end-to-end exactly-once behavior.
Common misconceptions
- A broker advertising exactly-once delivery guarantees exactly-once processing in every downstream system.
- Exactly-once means an event is never delivered twice on the network.
Anti-patterns
- Committing consumer offsets before the output write succeeds.
- Relying on duplicate detection without a stable, deterministic event key.
Related concepts
Related examples
Authoritative source
How it works
Exactly-once is usually built from at-least-once delivery plus a way to suppress the second effect. There are two common mechanisms. The first is idempotent processing: every event carries a stable identifier, and the sink records which identifiers it has applied, so a retry becomes a no-op. The second is transactional processing: the consumer writes output and commits its input offset in one atomic transaction, so a crash either preserves both or neither.
Stream processors add checkpoints. On restart, the job restores state from the last checkpoint and resumes from the corresponding offsets. If the sink participates in the checkpoint or supports transactional commits, the pipeline can avoid duplicates across failures. If it does not, the processor may replay events, and the sink must deduplicate.
The end-to-end condition
The guarantee must hold at every boundary: source, operator state, and sink. A common failure is a consumer that writes to a database and then commits the offset. If the process dies between those steps, the event is replayed and written twice. Reversing the order risks losing the event. Only a transaction spanning both, or an idempotent write keyed by event ID, closes the gap.
When to use it
Use exactly-once handling for money movement, inventory decrements, billing counters, and any effect that cannot be undone. For logs, metrics, or recomputable aggregates, at-least-once with deduplication at query time may be cheaper. Decide based on the cost of a duplicate, not on the elegance of the guarantee.
Trade-offs and example
Exactly-once adds coordination: transactions, unique constraints, deduplication tables, and longer commit paths. It can reduce throughput and increase latency. It also does not make side effects such as sending email reversible; external actions need their own idempotency keys.
Example: a consumer inserts (event_id, amount) into a ledger table with event_id as the primary key. A redelivered event violates the key and is ignored. This is simpler than a distributed transaction and gives the same visible result for that write.