Hinted Handoff
Hinted handoff lets a replica accept a write for an unavailable peer, store a hint, and replay it when that peer recovers, improving availability during partitions.
Definition
Hinted handoff is a replication technique in which a coordinator writes to a temporary stand-in when a preferred replica is unreachable. The stand-in stores a hint containing the intended target and the write, then delivers the data when the target returns. It preserves availability and reduces the need to block writes during short failures.
Context and scope
Dynamo-style stores use hinted handoff to keep writes available during node failures and network partitions. It is closely related to sloppy quorums, because the stand-in may be outside the preferred replica set.
Common misconceptions
- Hinted handoff makes a write durable on the intended replica immediately.
- Hints can be kept forever, so a long-offline node will always catch up without repair.
Anti-patterns
- Assuming hinted handoff alone guarantees consistency after a long partition.
- Letting the hint store grow without limits, monitoring, or an expiration policy.
Related concepts
Authoritative source
https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
How it works
A client sends a write to a coordinator. The coordinator normally forwards it to N replicas. If one preferred replica is down or unreachable, the coordinator writes to another healthy node and records a hint: the target node, the mutation, and a timestamp. The write can still meet its consistency level if enough replicas acknowledge it. When the target recovers, the hint is replayed, often with a delay to avoid overwhelming the returning node.
Hints have limits. They expire after a configured window, and a node that stays down longer than that window will miss writes. Hints also consume storage and replay bandwidth. If the hint store itself fails, the data must be recovered by anti-entropy repair.
Sloppy quorums
Hinted handoff often accompanies a sloppy quorum: the coordinator accepts writes on nodes outside the preferred set to maintain availability during a partition. A later read quorum may not include the node holding the hint, so a client can read stale data even when R + W > N appears satisfied. Strict quorums reject the write instead, preserving the overlap guarantee at the cost of availability.
When to use it
Use hinted handoff when availability during short, transient failures matters more than immediate consistency on every replica: shopping carts, session state, time-series ingestion. Do not rely on it as the only repair mechanism. Long outages, disk loss, or deleted hints require anti-entropy.
Trade-offs and example
The benefit is fewer failed writes and faster recovery for brief outages. The costs are extra storage, replay load, and a weaker consistency story. Hints can also arrive out of order, so the target must resolve conflicts with timestamps or version vectors.
Example: a three-replica keyspace with RF=3 and QUORUM writes. Node C is down, so the coordinator writes to node D and stores a hint for C. The write still gets two acknowledgements. When C returns, D replays the hint. If C stays down past the hint window, a scheduled repair must compare and reconcile the replicas.