Load Shedding
Load shedding rejects work early when a service is saturated, protecting latency and stability for the requests it accepts instead of failing everything.
Definition
Load shedding is an admission-control policy that refuses or defers work when demand exceeds capacity. Instead of accepting every request and letting queues grow, the service rejects some traffic early with an explicit overload response. Shedding can be based on concurrency, queue length, CPU, memory, or a priority class. The goal is to keep accepted requests within their latency target and prevent a transient spike from becoming a total outage.
Context and scope
Shedding is appropriate for services with a hard capacity limit and latency SLOs, such as APIs, gateways, and stream processors. It complements autoscaling, which reacts slowly, and rate limiting, which controls a client's long-term quota rather than instantaneous saturation.
Common misconceptions
- Shedding means crashing or dropping connections randomly — a good policy rejects predictably and tells the caller to retry
- Autoscaling makes shedding unnecessary — scaling takes seconds to minutes while overload arrives in milliseconds
- All requests have equal value — priority classes let interactive traffic survive while batch work is deferred
Anti-patterns
- Reading and parsing a large request before deciding to reject it
- Shedding only after every worker is blocked, so latency has already collapsed
- Returning a generic 500 instead of an overload signal with Retry-After
Related concepts
Related examples
Authoritative source
Protect the work you accept
A service that accepts every request under overload does not serve everyone; it serves everyone slowly. Queues grow, memory rises, timeouts fire, and clients retry, which adds even more load. Load shedding breaks that loop by limiting the number of requests in flight or waiting. The accepted requests keep their latency, and the rejected ones get a fast, explicit answer that lets the client back off.
The simplest form is a concurrency limit. When the limit is reached, new work is rejected immediately instead of waiting behind an unbounded queue:
export function createAdmission(maxInFlight: number) {
let inFlight = 0;
return async function admit<T>(work: () => Promise<T>): Promise<T> {
if (inFlight >= maxInFlight) {
const error = new Error('overloaded');
(error as Error & { status?: number }).status = 503;
throw error;
}
inFlight += 1;
try {
return await work();
} finally {
inFlight -= 1;
}
};
}This snippet assumes a single process. In a multi-instance deployment, each instance applies its own limit, and a gateway may apply a global one. The right limit is derived from measured concurrency at the latency target, not from a guess. Start with the number of workers or database connections the service can actually use, then load test.
Choose a signal
Concurrency is a good default because it directly bounds resource usage. Queue length catches the case where work waits before running. CPU or event-loop lag detects saturation that concurrency alone may miss, especially for CPU-bound handlers. Memory pressure can trigger shedding before the process is killed. Combine signals with hysteresis: a single spike should not flap the policy on and off.
Priority classes matter. A checkout request and a nightly report should not compete equally. The checkout path can reserve capacity, while batch work is rejected or delayed when the interactive class is near its limit. Priorities must be authenticated, not taken from a client-supplied header.
Reject with the right response
Use 503 Service Unavailable with a Retry-After header for temporary overload, and 429 Too Many Requests when a specific client exceeds its quota. The distinction matters: 429 is about the caller’s rate, 503 is about the server’s capacity. Include a short, machine-readable body and log the shed count with the reason.
Clients must retry with exponential backoff and jitter. A client that retries immediately turns a brief overload into a sustained one. If the request is not idempotent, retries need an idempotency key so a shed that happens after partial work does not duplicate side effects.
Trade-offs and verification
Shedding makes some requests fail so the rest can succeed. That is a product decision, not only an infrastructure one. Define which classes may be shed, what the user sees, and how the system recovers. Test the policy with a load generator that exceeds capacity: accepted-request p95 should stay within the target, rejected requests should return quickly, and the service should recover without a restart when load drops. Monitor shed rate, in-flight count, queue depth, and latency together; a shed rate near zero during a latency spike means the policy is sampling too late.
When to use it
Use load shedding when a service has a hard concurrency limit and a latency SLO that must survive traffic spikes. The benefit is bounded resource use; the drawback is that some callers receive an error by design. Compared to rate limiting, shedding reacts to current server saturation rather than a client’s long-term quota. Compared to autoscaling, it acts in milliseconds instead of minutes. The two mechanisms complement each other rather than compete.