Retry Budget

A retry budget caps retries as a fraction of successful traffic so that retry storms cannot amplify load during an outage.

resilience

Definition

A retry budget is a shared limit on how much retry traffic a system may generate relative to its successful traffic. Instead of allowing every request to retry independently, the system tracks a budget and only permits a retry when the budget has capacity. This prevents retry amplification from turning a partial outage into a total one.

Context and scope

Retries improve success rates for transient failures, but each retry multiplies load. If a dependency is already failing, independent retries from many clients can double or triple traffic, delaying recovery. A retry budget bounds the extra load and forces callers to fail fast when the system is unhealthy.

Common misconceptions

More retries always improve reliability — retries help only when failures are transient and capacity exists
A per-request retry limit is a retry budget — a per-request limit still multiplies across all clients; a budget is shared and traffic-aware
Retry budgets replace circuit breakers — budgets limit retry volume; breakers stop calls to unhealthy dependencies

Anti-patterns

Retrying every 5xx immediately with no backoff or jitter
Retrying non-idempotent writes without an idempotency key
Setting a budget so large that it never constrains anything during an outage

Related concepts

Related examples

Authoritative source

https://sre.google/sre-book/handling-overload/

Retry amplification

Suppose 10% of requests fail and every request retries twice. The failed 10% becomes an extra 20% of traffic precisely when the dependency is least able to serve it. If the dependency’s failure rate rises, retries can push a degraded service into collapse. A retry budget makes the extra traffic a controlled fraction of successes.

A simple budget

Track successes and retries in a rolling window. Permit a retry only while retries < successes * ratio. With a 10% ratio and 1,000 successes per minute, at most 100 retries per minute are allowed.

class RetryBudget {
  constructor(ratio = 0.1, windowMs = 60_000, now = Date.now) {
    this.ratio = ratio;
    this.windowMs = windowMs;
    this.now = now;
    this.successes = [];
    this.retries = [];
  }

  recordSuccess() {
    this.prune();
    this.successes.push(this.now());
  }

  tryRetry() {
    this.prune();
    if (this.retries.length >= this.successes.length * this.ratio) {
      return false;
    }
    this.retries.push(this.now());
    return true;
  }

  prune() {
    const cutoff = this.now() - this.windowMs;
    this.successes = this.successes.filter((t) => t >= cutoff);
    this.retries = this.retries.filter((t) => t >= cutoff);
  }
}

Backoff and deadlines

A budget controls volume, not timing. Pair it with exponential backoff and jitter, and enforce a total deadline so retries cannot extend a request indefinitely. The retry budget with deadline example combines these concerns.

When to use it

Use a retry budget for client libraries and service meshes that call shared dependencies. Compared to a per-request retry count, a budget is adaptive to overall traffic. Compared to a circuit breaker, a budget reduces retry load but still allows calls; the breaker stops calls once failure is confirmed.

Pitfalls

Budgets require shared state or per-instance accounting; per-instance budgets approximate the global limit and can still amplify under fan-out. Never retry non-idempotent operations without an idempotency key. Measure both retry attempts and the final success rate; a budget that improves success rate by hiding persistent failures is not working.

Retry budget with total deadline