Diagnose API Requests That Hang Forever

Spot requests stuck without timeouts, count half-open connections, find the dependency with no deadline, and add end-to-end request deadlines.

~35 minapiresiliencedebugging

Task

Find which dependency swallows stuck requests and add deadlines so the API fails fast instead of accumulating half-open connections.

When to use this

Every few days the API stops responding entirely. CPU is idle, memory is fine, and restarts fix it. Thread dumps show hundreds of requests waiting on the same downstream call.

Prerequisites

Access to runtime introspection, such as node --inspect or jstack
The ability to replay a stuck scenario against a non-production environment
A list of external calls the service makes

Steps

1. Count open sockets toward the suspect dependency.

ss -tn state established "( dport = :443 )" | grep -c <payment-ip>

Expected output: Hundreds of established sockets to one host while the service is idle on CPU is the signature of requests stuck in a read with no deadline.

2. Take a snapshot of what the stuck handlers are doing.

kill -USR1 <pid> && tail -100 /var/log/app/diagnostics.log

Expected output: A diagnostic dump lists in-flight requests with their phase; all of them sit at awaiting response from payments.

3. Reproduce the hang deterministically with a dead endpoint.

PAYMENT_BASE_URL=http://10.255.255.1:9000 npm run reproduce:hang

Expected output: The black-hole address accepts connections but never replies, and the request hangs indefinitely, proving no timeout exists.

4. Add a deadline on the whole call chain.

const res = await fetch(`${PAYMENT_BASE_URL}/v1/charge`, {
  signal: AbortSignal.timeout(3000),
});

Expected output: The reproduced hang now aborts after three seconds with a TimeoutError instead of blocking a handler forever.

5. Propagate deadlines instead of stacking independent timeouts.

async function charge(ctx, payload) {
  return fetch(`${PAYMENT_BASE_URL}/v1/charge`, {
    signal: ctx.deadline, // derived from the incoming request budget
  });
}

Expected output: When the caller has two seconds left, the downstream call gets at most two seconds, so retries cannot exceed the user-facing budget.

Troubleshooting

Problem

Timeout added but pool still exhausts

Solution

The HTTP client retries the timed-out call with no cap; bound retries by total elapsed time or attempts so one slow dependency cannot consume the pool.

Problem

Timeout fires but sockets linger

Solution

The abort did not destroy the connection; ensure the client honors abort by destroying the socket, and set a keep-alive idle cap on the pool.

Problem

Hang only appears in production

Solution

Production-only dependencies such as a slow partner endpoint are absent locally; reproduce with a black-hole address and assert on deadline behavior in CI.

Limitations

Deadlines trade waiting for errors; the user still fails unless a fallback path exists.
Per-call timeouts are necessary but not sufficient; without deadline propagation the sum of stages can exceed any single budget.

Idle CPU is not health

A service that hangs rarely burns resources: it accumulates requests parked on a read that never completes. The evidence is in the socket table, not the CPU graph. Counting established sockets to one dependency, then snapshotting what in-flight handlers wait on, names the culprit. The permanent fix is a deadline on every external call, and ideally one deadline derived from the caller budget and propagated through the chain.

# Requests parked on one dependency
ss -tn state established "( dport = :443 )" | grep -c <payment-ip>

# Reproduce a hang with a black-hole address (never responds)
PAYMENT_BASE_URL=http://10.255.255.1:9000 npm run reproduce:hang

One budget, propagated

Independent per-call timeouts multiply: a 3-second call, retried twice inside a 5-second handler, blocks a worker for 9 seconds. Derive the downstream signal from the incoming request remaining time so retries draw from the same budget, and the whole chain fails within what the user already agreed to wait.

Fail fast, then degrade deliberately

A deadline converts an invisible hang into a fast, loggable error, which is the precondition for graceful degradation: serve cached data, queue the work, or show a partial view. Without the deadline you get neither the error nor the fallback, only a restart at 3 a.m.