HTTP 429 — Retry Storm Makes Rate Limiting Worse

Handle 429 responses without amplifying load: honor Retry-After, add jittered exponential backoff, deduplicate callers, and shed load before the limit.

intermediatedebugginghttprate-limitingresilience

Symptoms

A brief upstream 429 turns into minutes of escalating errors as every client retries simultaneously
Error rate spikes sawtooth: recovery begins, then a synchronized retry wave triggers 429 again
Retry-After headers exist in responses but client logs show retries within milliseconds

Possible causes

All failing callers retry after a fixed delay, synchronizing their attempts into a wave
Retry-After is ignored, so retries land immediately during the busiest window
Each layer (gateway, service, client SDK) retries independently, multiplying the original request
Retries have no budget: the same request retries dozens of times, each competing with new traffic

Diagnosis steps

Read the Retry-After semantics in responses

curl -sI https://api.example.com/v1/orders | grep -iE "retry-after|ratelimit"

Interpretation

The header value (seconds or HTTP-date) is the server-declared safe re-entry time; ignoring it is the root of the storm.

Measure retry amplification

grep "attempt=" access.log | awk '{print $NF}' | sort | uniq -c

Interpretation

A distribution with heavy tails (10+ attempts) means no retry budget; attempts count per request should cap low.

Check clock alignment across layers

grep -Rn "retry\|backoff" gateway-config/ src/middleware/

Interpretation

Independent retry policies at multiple layers multiply: gateway 2 retries x service 3 retries x SDK 3 retries equals 18 upstream attempts.

Root cause

Synchronized retries convert a transient rate-limit response into a sustained overload: when the limit triggers, all rejected requests share the same recovery timing, and their simultaneous retry re-triggers the limit, cycling the system between overload and recovery instead of converging.

Fix

Honor Retry-After with jittered exponential backoff (full jitter), cap attempts per request with a retry budget, retry only idempotent operations or attach idempotency keys, collapse retries to a single layer (usually closest to the failure), and shed or queue load client-side before hitting the limit.

Risk note

Retrying non-idempotent operations after 429 can duplicate writes when the server did process the request. Use idempotency keys or do not retry.

Verify the fix

Under an upstream-degraded scenario, retry waves disappear from metrics: upstream request rate stays flat, error convergence is monotonic, and Retry-After compliance shows in client timing logs.

Symptom walkthrough

The sawtooth in the error-rate graph is the signature: errors climb, recovery starts, then a synchronized wave of retries spikes them again. The response headers hold the contract being ignored:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Remaining: 0

Count the amplification next: attempts-per-request distribution with a heavy tail (10 or more attempts) means no budget, and multi-layer retry stacks (gateway x service x SDK) multiply a single logical request into a dozen upstream calls, all sharing the same fixed backoff and therefore the same wake-up time.

Fix in practice

Full-jitter exponential backoff anchored on the server value:

async function callWithRetry(url, opts) {
  const maxAttempts = 4;
  let attempt = 0;
  while (true) {
    const res = await fetch(url, opts);
    if (res.status !== 429) return res;
    if (++attempt >= maxAttempts) throw new RateLimitError(attempt);

    const serverHint = Number(res.headers.get('retry-after')) || 0;
    const cap = Math.max(serverHint, 2 ** attempt);   // never below server hint
    const delay = Math.random() * cap;                 // full jitter
    await sleep(delay);
  }
}

Collapse retries to one layer — the one closest to the failure — and give every request a budget:

retry policy: max 3, jittered, idempotent methods only (or Idempotency-Key header)