Add Load-Shedding Middleware

Reject new work with 503 and Retry-After when in-flight requests exceed a measured limit, keeping accepted requests within their latency target.

~25 minresiliencenodejscapacity

Task

Prevent an overload spike from collapsing latency for every request by bounding concurrency and failing fast at the admission boundary.

When to use this

A traffic spike or a slow dependency increases in-flight requests until queues and memory grow. Every caller slows down, and retries make the load worse.

Prerequisites

An HTTP service with a measured concurrency limit
A load test that identifies the limit at the latency target
A reverse proxy that preserves the 503 response and Retry-After header

Steps

1. Add an admission counter that rejects when the in-flight count reaches the limit.

export function loadShedding(maxInFlight: number, retryAfterSeconds = 5) {
  let inFlight = 0;

  return (req, res, next) => {
    if (inFlight >= maxInFlight) {
      res.setHeader('Retry-After', String(retryAfterSeconds));
      return res.status(503).json({ error: 'overloaded' });
    }
    inFlight += 1;
    res.on('finish', () => { inFlight -= 1; });
    res.on('close', () => { inFlight -= 1; });
    next();
  };
}

Expected output: The request over the limit receives 503 immediately, and accepted requests decrement the counter on finish or close.

2. Mount the middleware before body parsing and expensive work, and exclude health probes.

app.use('/healthz', healthHandler);
app.use(loadShedding(Number(process.env.MAX_IN_FLIGHT ?? 100)));
app.use(express.json());

Expected output: Overload is rejected before JSON parsing or database work.

3. Load test beyond the limit and verify accepted-request p95 and rejection speed.

hey -z 30s -c 200 http://localhost:3000/api/items

Expected output: Accepted requests stay near the target latency and rejected requests return quickly with 503.

Troubleshooting

Problem

In-flight count goes negative

Solution

Both finish and close can fire for one request. Guard the decrement with a per-request flag or use a single termination event.

Problem

Requests are shed before the service is actually saturated

Solution

The limit is too low or the measurement excluded queueing. Derive it from a load test at the SLO, not from CPU alone.

Limitations

The counter is per process; a global limit needs a gateway or shared state.
Shedding is a product decision: some requests fail by design to protect the rest.

Bound what you accept

A queue that accepts everything under overload serves everyone slowly. Load shedding bounds in-flight work and rejects the excess at the admission boundary. The accepted requests keep their latency; the rejected ones receive a fast, explicit signal.

if (inFlight >= maxInFlight) {
  res.setHeader('Retry-After', '5');
  return res.status(503).json({ error: 'overloaded' });
}
inFlight += 1;
res.on('finish', () => { inFlight -= 1; });

Measure the limit

Run a load test and find the concurrency at which p95 starts to exceed the target. That is the starting limit, not the number of CPUs. Include queueing and dependency latency. Revisit after changing the query path, instance size, or downstream capacity.

Reject correctly

Use 503 with Retry-After for server overload; use 429 for a client quota. Log the shed count and reason, but do not log every rejected request at error level. Exclude health checks so the orchestrator can still observe the process.

Verify the limit

Expected outcome: under a load test above the limit, accepted requests stay near the latency target and excess requests receive 503 quickly with a positive Retry-After. Confirm that the in-flight counter returns to zero after the test, including when clients disconnect mid-request. If the counter drifts upward, the middleware is registering more than one decrement or missing a termination event.