Inject Timeouts at the Dependency Boundary

Make a downstream call hang in a controlled way, then assert the timeout path returns 504, cancels the request, and releases resources.

~35 mintestingresiliencetimeouts

Task

Prove that a slow dependency triggers the timeout handler instead of hanging the request or leaking a connection.

When to use this

A payment provider stops responding. Without a timeout the request waits until the client disconnects, the worker stays busy, and the pool drains during the incident.

Prerequisites

Node.js 22 with the built-in fetch (undici)
Toxiproxy or a stub server that can delay responses
A test database or pool whose state can be asserted

Steps

1. Put an explicit timeout on every outbound call.

const res = await fetch(url, {
  method: 'POST',
  body: JSON.stringify(payload),
  signal: AbortSignal.timeout(2000),
});

Expected output: The call rejects with a TimeoutError after about two seconds instead of waiting for the OS default.

2. Map the timeout to a gateway response and do not silently retry a non-idempotent request.

try {
  return await chargeCard(order);
} catch (err) {
  if (err.name === 'TimeoutError' || err.name === 'AbortError') {
    return res.status(504).json({ error: 'upstream_timeout', retryable: false });
  }
  throw err;
}

Expected output: Clients receive 504 with retryable false for a charge, preventing duplicate payments.

3. Inject a deterministic hang with a Toxiproxy latency toxic.

toxiproxy-cli create payment -l 127.0.0.1:8666 -u payment-provider:443 && toxiproxy-cli toxic add payment -t latency -a latency=5000

Expected output: Requests through the proxy take at least five seconds, so the two-second client timeout always wins.

4. Assert the response time and that no pooled resource stays checked out.

const started = Date.now();
const res = await request('/api/checkout', { method: 'POST', body: order });
assert.equal(res.status, 504);
assert.ok(Date.now() - started < 3000);
assert.equal(pool.totalCount, pool.idleCount);

Expected output: The response arrives under three seconds and every database client is back in the pool.

5. Remove the toxic and verify that the next request recovers without a restart.

toxiproxy-cli toxic remove payment -n latency_downstream && curl -s -o /dev/null -w '%{http_code}\n' -X POST http://localhost:3000/api/checkout

Expected output: 200 on the next request, proving the failure was bounded and the service recovered on its own.

Troubleshooting

Problem

The timeout never fires

Solution

No signal was passed; Node's fetch has no default timeout. Also check that a proxy or gateway in front is not buffering or waiting longer.

Problem

The endpoint returns 500 instead of 504

Solution

A generic error handler catches AbortError before the timeout mapping. Handle timeout errors explicitly before the catch-all middleware.

Problem

The customer is charged twice after a timeout

Solution

The upstream may have processed the request even though the response was lost. Send an idempotency key and treat timeouts as unknown outcomes, not failures.

Limitations

A client-side timeout does not cancel work already accepted by the upstream service.
Network timeouts and application timeouts are different layers; test both and keep them consistent.
Toxiproxy adds a network hop and its own latency, so verify the configured delay in the real environment.

The default is to wait forever

Node’s fetch has no request timeout. A dependency that accepts the connection and never responds will hold the request until the socket layer gives up, which can be minutes. During that time the worker is occupied, the database client is checked out, and the client has probably retried. One slow dependency becomes a self-inflicted outage because nothing bounded the wait.

Bound it at the call site

Pass an explicit AbortSignal.timeout() on every outbound request, chosen from the dependency’s latency budget rather than a global default. A payment provider that normally answers in 300 ms does not need 30 seconds. When the signal fires, the promise rejects with TimeoutError, and the handler must map it to a gateway error rather than letting it become a 500.

const res = await fetch(url, { signal: AbortSignal.timeout(2000) });

Decide what a timeout means

For a read, a timeout is a retryable failure. For a charge, it is an unknown outcome: the request may have succeeded and the response was lost. Return 504 with an explicit flag, do not auto-retry, and record the operation so it can be reconciled. This is where timeout handling and idempotency meet; testing one without the other leaves a gap that shows up as duplicate charges.

Inject the hang, do not hope for one

Waiting for a real dependency to slow down makes tests flaky and useless. Toxiproxy can add a latency toxic that delays responses by five seconds, which is deterministic and removable. A simple stub that accepts the connection and never writes is even more direct. The important property is that the delay is longer than the client timeout by a comfortable margin, so the test measures the timeout, not a race.

Assert resources, not just status

A 504 response is only half the proof. If the handler leaked the database client or left a timer running, the next incident will be a pool exhaustion rather than a slow dependency. Assert that pool.totalCount === pool.idleCount after the request, and that the process has no pending handles. Then remove the fault and confirm the service recovers without a restart, which proves the failure did not leave permanent state.