Thundering Herd

A thundering herd is a synchronized surge of retries, reconnects, or cache misses that overwhelms a recovering service and delays recovery further.

resilience

Definition

A thundering herd occurs when many clients or workers react to the same event at the same time. Examples include every client retrying after an outage, every process reconnecting after a network partition, or every cached entry expiring on the same schedule. The synchronized load can exceed the capacity of a service that has just recovered, causing another failure and another synchronized retry wave.

Context and scope

Herds appear in retry policies, connection pools, cache TTLs, cron jobs, and distributed locks. They are most dangerous during recovery, when capacity is reduced and the system is least able to absorb a spike.

Common misconceptions

Retrying immediately improves availability — a synchronized retry storm can keep a service down
Any random delay is enough — jitter must be applied per attempt and per client, with a bounded cap
The herd only affects the failing service — it also consumes connection pools and threads in callers

Anti-patterns

Fixed retry intervals shared by every client
Refreshing every cache key at the top of the hour
Reconnecting all workers with the same backoff schedule after a broker restart

Related concepts

Related examples

Authoritative source

https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

Why synchronization is dangerous

A service fails at 10:00:00. Ten thousand clients have a one-second retry timer. At 10:00:01 they all retry, the service receives the same load that caused the failure, and it fails again. The pattern repeats until someone intervenes. The clients are behaving reasonably in isolation; their schedules are the problem.

The same effect appears without failures. A cache with a one-hour TTL populated at startup expires every entry at once. A cron job that recomputes a shared report at midnight wakes every worker. A lock service that releases a lock after a timeout lets all waiters race to acquire it. In each case, coordination creates a spike that capacity planning did not anticipate.

Break the synchronization

Retries should use exponential backoff with jitter. The delay grows with each attempt, and a random component spreads clients across time:

import random
import time


def backoff_delay(attempt: int, base: float = 0.2, cap: float = 10.0) -> float:
    exponential = min(cap, base * (2 ** attempt))
    return random.uniform(0, exponential)  # full jitter

Full jitter samples uniformly between zero and the exponential ceiling. It is simple and effective, at the cost of occasionally retrying quickly. Equal jitter, which keeps half the delay fixed and randomizes the other half, trades some spread for a minimum wait. Whichever policy you choose, apply it to every attempt and stop when the retry budget or deadline is exhausted.

For caches, add jitter to TTLs and refresh hot keys in the background before they expire. For locks, add randomized wait before retry and use a fair queue where possible. For connection pools, reconnect with jitter and limit the reconnect rate per process.

Avoid a herd on recovery

Recovery is the worst time for a spike. A circuit breaker that opens for everyone at once, a load balancer that returns all traffic to a recovered instance, or a queue that releases all delayed messages simultaneously can re-break the system. Ramp traffic gradually, use readiness gates, and keep concurrency limits in place during the ramp.

Track retry attempts, reconnect rate, and cache-miss bursts as first-class metrics. A rising retry rate during an incident is a signal that the backoff policy is not working or that clients are ignoring it. The goal is not to eliminate retries; it is to make them a controlled trickle rather than a synchronized wave.

When to use it

Apply jitter whenever many clients can react to the same event: retry policies, reconnect loops, cache refreshes, and scheduled jobs. The advantage is a smooth load profile during recovery; the drawback is a slightly longer average wait for an individual caller. Compared to fixed exponential backoff, jitter sacrifices determinism for stability. Compared to a shared retry service, it is simpler and requires no coordination infrastructure.