Add Latency Fault Injection for Resilience Tests

Inject bounded latency behind an explicit test flag, verify timeouts and fallbacks, and guarantee the injection is disabled outside test environments.

~30 minresiliencetestingoperations

Task

Test how the service behaves when a dependency or handler is slow, without changing production behavior or leaving a permanent fault path enabled.

When to use this

Timeouts and fallbacks exist in code but have never been exercised. A slow dependency causes a cascading failure in production.

Prerequisites

A non-production environment or a canary with explicit authorization
A configuration flag that is off by default and cannot be enabled accidentally
A test that asserts the timeout response and fallback behavior

Steps

1. Add a latency injection middleware guarded by an environment flag and a secret test header.

app.use(async (req, res, next) => {
  if (process.env.NODE_ENV === 'production') return next();
  if (process.env.FAULT_INJECTION !== '1') return next();
  if (req.header('x-fault-latency') !== process.env.FAULT_TOKEN) return next();
  await new Promise((resolve) => setTimeout(resolve, Number(req.header('x-fault-ms') ?? 1500)));
  next();
});

Expected output: Only an explicitly authorized test request is delayed; normal requests are unaffected.

2. Set the request timeout shorter than the injected delay and assert the timeout response.

await request(app)
  .get('/api/slow')
  .set('x-fault-latency', process.env.FAULT_TOKEN)
  .set('x-fault-ms', '2000')
  .expect(503);

Expected output: The request returns the documented timeout response within the configured budget.

3. Verify the flag-off path and remove the injection from production builds or ensure the guard cannot be enabled.

NODE_ENV=production FAULT_INJECTION=1 node server.js

Expected output: Production mode always bypasses the injection.

Troubleshooting

Problem

The fault path is enabled in production

Solution

Guard on the environment and a secret token, fail closed, and add a test that production mode bypasses the middleware.

Problem

The delay blocks the event loop

Solution

Use a timer or an async sleep, not a busy loop; injected latency should simulate I/O, not CPU starvation.

Limitations

In-process injection cannot simulate network partitions or DNS failures.
Fault injection must be authorized and scoped; it is a test tool, not a production feature.

Simulate the slow dependency

Timeouts and fallbacks are easy to write and easy to leave untested. A latency injection lets a test slow a request and observe whether the deadline, fallback, and error response work. Keep the delay bounded and the injection behind an explicit flag and token.

if (process.env.NODE_ENV === 'production') return next();
if (req.header('x-fault-latency') !== process.env.FAULT_TOKEN) return next();

Fail closed

The injection must be impossible to enable accidentally in production. Check the environment first, default the flag to off, and require a secret header. Add a test that production mode always bypasses the middleware. A permanent debug delay is a latent outage.

Test the whole response

Assert the status code, the error body, the elapsed time, and the fallback behavior. Also test that the dependency call was cancelled rather than continuing in the background. The point is not the delay itself; it is whether the system’s failure handling works under it.

Verify the injection boundary

Expected outcome: an authorized test request with a delay longer than the deadline returns the documented timeout response, while a normal request and a production-mode request are unaffected. Confirm with a test that sets the fault headers, a test without them, and a test with NODE_ENV=production and the flag enabled. The production test must prove the middleware bypasses the delay.