Inject an Error Rate Into a Dependency

Return a controlled share of 500s from a fake dependency, then assert retry limits, circuit breaking, and graceful degradation.

~35 mintestingchaosresilience

Task

Verify that a partial dependency failure degrades gracefully instead of becoming a full outage or a retry storm.

When to use this

A search service starts failing thirty percent of requests. Without retry limits and a circuit breaker, each failed request retries three times and multiplies load on a struggling dependency.

Prerequisites

Node.js 22 and a local stub HTTP server
opossum or an equivalent circuit breaker library
A cache or fallback path for degraded responses

Steps

1. Inject failures deterministically so exactly thirty percent of calls fail.

let call = 0;
let failing = true;

function flakyHandler(req, res) {
  if (req.url === '/faults/off') {
    failing = false;
    res.writeHead(204).end();
    return;
  }
  call += 1;
  if (failing && call % 10 < 3) {
    res.writeHead(500, { 'content-type': 'text/plain' }).end('injected');
    return;
  }
  res.writeHead(200, { 'content-type': 'application/json' }).end('{"ok":true}');
}

Expected output: Every tenth-call window contains three failures, and calling /faults/off stops injection without a restart.

2. Limit retries to one extra attempt with jitter instead of a fixed three-retry loop.

const res = await retry(() => search(q), {
  attempts: 2,
  baseDelayMs: 50,
  jitter: true,
});

Expected output: A failing call produces at most two upstream requests, not four, so the dependency is not amplified.

3. Wrap the dependency in a circuit breaker with a volume threshold and a short reset timeout.

import CircuitBreaker from 'opossum';

const breaker = new CircuitBreaker(search, {
  timeout: 1000,
  errorThresholdPercentage: 50,
  volumeThreshold: 10,
  resetTimeout: 5000,
});

Expected output: After at least ten calls with more than half failing, the breaker opens and later calls reject immediately.

4. Serve a degraded response while the breaker is open instead of propagating the error.

try {
  return res.json(await breaker.fire(q));
} catch (err) {
  if (err.code === 'EOPENBREAKER' || err.code === 'ETIMEDOUT') {
    res.set('Warning', '110 - "Response is Stale"');
    return res.json({ results: cache.get(q) ?? [], degraded: true });
  }
  throw err;
}

Expected output: Clients receive 200 with degraded true and a Warning header, and the error rate stays within the SLO.

5. Stop the fault, wait for the reset timeout, and confirm the breaker closes after a successful probe.

curl -s -X POST http://localhost:9999/faults/off && sleep 6 && curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:3000/api/search?q=books'

Expected output: 200 from the real dependency again, with the breaker state back to closed and no restart required.

Troubleshooting

Problem

Retries make the outage worse

Solution

Cap attempts, add jitter, and use a retry budget shared across the fleet. Never retry non-idempotent writes or 4xx responses.

Problem

The circuit breaker never opens

Solution

Errors are being swallowed and returned as 200, or the volume threshold was never reached. Count failures at the dependency boundary.

Problem

The test passes locally but flakes in CI

Solution

Use the deterministic modulo injector, shorten resetTimeout, and avoid asserting wall-clock timing around the half-open state.

Limitations

Synthetic faults cannot reproduce every real failure mode, such as partial responses or slow bodies.
Breaker state is per process; a fleet may have some instances open and others closed.
Degradation only works if a correct fallback exists; the injector proves the path runs, not that the data is right.

Partial failure is the hard case

A dependency that is completely down is easy: the breaker opens, traffic stops, and everyone understands. A dependency failing thirty percent of the time is harder, because every request looks like it might succeed. Retries feel reasonable, but three retries on a thirty percent failure rate turn one user request into four upstream calls and push a struggling service over the edge. The failure mode to design for is degradation, not perfection.

Make the failure rate exact

Random failure injection produces flaky tests and unreproducible results. A modulo counter, or a seeded random generator, gives an exact rate and a deterministic sequence. The handler returns 500 for three out of every ten calls, which is enough to trip a fifty percent breaker after the volume threshold without depending on timing. When debugging, the same sequence replays identically.

if (call % 10 < 3) return res.writeHead(500).end('injected');

Budget retries before adding a breaker

A breaker without a retry budget still allows bursts: the first failures retry, then the breaker opens. Decide the maximum upstream calls per user request, count them, and make retries consume that budget. Jitter prevents synchronized retries from every instance. For writes, retries need idempotency keys; a timeout is not proof of failure.

Degrade with something useful

An open breaker should not mean a blank page. Serve cached results, partial data, or a clearly labeled fallback, and tell the client with a Warning header or a degraded field. The status code should reflect what the client can do: a usable degraded response is a 200, while a request that cannot be answered at all is a 503. Document which endpoints have fallbacks and which fail closed.

Verify recovery, not just failure

The half-open state is where many implementations break: the breaker allows one probe, the probe succeeds, and traffic floods back into a dependency that is still warming up. Test that the breaker closes after a successful probe, and consider a gradual ramp instead of an immediate reset. The recovery test is as important as the failure test, and it is the one teams usually skip.