Hot Key Rebalancing Workflow
Identify a hot key that overloads one shard, split it across replicas or sub-keys, and verify the load is balanced without changing the application semantics.
Objective
Relieve the load on a hot key by splitting it across multiple replicas or sub-keys, so no single shard is a bottleneck, without changing what the application reads or writes.
Environment
- A sharded store with per-shard load metrics
- A hot key identified by disproportionate load
- A way to add replicas or sub-keys without a migration
Steps
1. Confirm the hot key
Check the per-shard load and confirm one shard is disproportionately loaded by one key.
Expected state: A hot key and its shard, with the load ratio to the average.
2. Choose a split strategy
If the key is read-heavy, add replicas and load-balance reads; if write-heavy, split the key into sub-keys by a hash of the value.
Expected state: A strategy that splits the load without changing the application semantics.
3. Apply the split
Add the replicas or sub-keys and route the traffic through a layer that distributes the hot key access.
Expected state: The hot key access is distributed across replicas or sub-keys.
4. Verify balance
Run the hot-key traffic and assert the per-shard load is within a band of the average, and the application semantics are unchanged.
Expected state: The load is balanced and the application behaves the same.
Decision points
Replicas or sub-keys?
- Replicas for read-heavy
- Sub-keys for write-heavy
Replicas for read-heavy, because writes to replicas are serialized; sub-keys for write-heavy, because each sub-key is independently writable.
How many splits?
- 2x the load ratio
- 10x for headroom
2x the load ratio, enough to balance without over-sharding.
Expected outcome
The hot key no longer overloads one shard, the per-shard load is balanced, and the application reads and writes the same data.
Verification
Run the hot-key traffic for 10 minutes and confirm the per-shard load is within 20% of the average, and the application output is identical to before the split.
Cleanup
Keep the replicas or sub-keys and the routing layer; remove the diagnostic metrics.
Confirm the hot key
A hot key is one key that receives a disproportionate share of the traffic, overloading its shard while the others are idle. The per-shard load metrics show the imbalance, and the top key per shard identifies the hot key.
metrics shard.load | sort -rn | head -1
top-key --shard $HOT_SHARDChoose a split strategy
A read-heavy hot key is split by adding replicas and load-balancing reads across them. A write-heavy hot key cannot be replicated, because writes must serialize, so it is split into sub-keys by a hash of the value, and the application reads and writes the sub-keys through a routing layer.
if read_heavy: add-replicas $KEY --count $N
if write_heavy: split-key $KEY --by hash --count $NApply the split and route
Add the replicas or sub-keys and route the hot key access through a layer that distributes it. For replicas, a round-robin or a hash of the caller distributes reads. For sub-keys, the routing layer hashes the value to pick the sub-key, so the application sees one key but the store sees N.
const sub = hash(value) % N;
store.get(`${key}:${sub}`);Verification
The hot-key traffic runs for 10 minutes with the per-shard load within 20% of the average, and the application output is identical to before the split. The split is transparent to the application, which reads and writes the same logical key.