Node.js — SIGTERM Kills In-Flight Requests During Deploys

Fix 502 bursts during rolling deploys by handling SIGTERM properly: stop accepting, close idle keep-alive connections, await in-flight work, then exit within budget.

intermediatedebuggingnodejsdeploymentshutdown

Symptoms

A small burst of 502 or connection reset errors appears in the load balancer logs at every deploy
Pods or VMs terminate 10 seconds after SIGTERM while requests were still processing
server.close() was called but the process hangs because keep-alive sockets never drain

Possible causes

No SIGTERM handler exists, so the default action terminates the process immediately mid-request
server.close() waits for keep-alive connections to end naturally, which can outlive the termination grace period
In-flight async work (queues, timers, database commits) is not awaited before exit
A health endpoint keeps returning healthy during shutdown, so the load balancer keeps sending new traffic

Diagnosis steps

Watch the termination sequence

kubectl logs pod/app-7d9c6 -f | grep -E "SIGTERM|close|exit"

Interpretation

A log gap between SIGTERM and process exit with no drain messages shows the shutdown path is doing nothing or hanging.

Count open sockets at shutdown

process.on('SIGTERM', () => log({ conns: server._connections }));

Interpretation

A high connection count that does not shrink after server.close points at lingering keep-alive sockets.

Reproduce locally with a slow request

node server.js & sleep 1; curl -m 30 localhost:3000/slow & kill -TERM %1

Interpretation

A reset curl response reproduces the deploy-time failure in isolation, before any orchestrator is involved.

Root cause

SIGTERM arrives while the server still serves traffic. Without an explicit handler the process dies instantly; with a naive server.close() it hangs on keep-alive sockets past the grace period and is then SIGKILLed. Both endings drop in-flight requests, which the load balancer reports as 502s during every deploy.

Fix

Implement a full sequence: mark unhealthy so the balancer stops routing, stop listening, force-close idle keep-alive sockets with closeIdleConnections, await a completion signal (counter drained or a fixed deadline), close resource pools, then exit with status 0 inside the grace budget.

Risk note

Waiting for unbounded work past the grace period guarantees SIGKILL and loses the same requests you tried to save. Always race the drain against a deadline.

Verify the fix

Run a rolling deploy under constant load and assert zero 502s, all in-flight requests completed, and termination time reliably under the grace period.

Symptom walkthrough

Deploy-time 502 bursts follow a strict rhythm: they start exactly when the first pod terminates and last as long as the rollout. Locally you can shrink the whole story to three commands:

node server.js &
curl -m 30 localhost:3000/slow &   # in-flight during shutdown
kill -TERM %1                        # SIGTERM mid-request

A reset curl reproduces production with zero orchestrators involved. The other half of the story is the hang: server.close() only stops new connections and waits for existing ones — including keep-alive sockets that stay open for minutes — so the process blows the grace period and receives SIGKILL.

Fix in practice

A correct shutdown is a sequence, not a single call:

let shuttingDown = false;

app.use((req, res, next) => {
  if (shuttingDown) res.setHeader('Connection', 'close');
  next();
});

process.on('SIGTERM', async () => {
  shuttingDown = true;
  server.close();               // stop accepting
  server.closeIdleConnections(); // kill keep-alive idlers (Node 18.2+)
  await Promise.race([
    drainInFlight(10_000),      // your counter or tracker
    delay(20_000),              // deadline under the 30s grace
  ]);
  await closePools(db, redis, mq);
  process.exit(0);
});

The health endpoint must return failing as soon as shuttingDown flips, so the balancer and readiness probe stop routing to this instance immediately. Responses during drain should send Connection: close so clients do not reuse the dying socket.

Verification

Run a real rolling deploy while a constant-load generator streams traffic. Assert three metrics from the balancer: 502 count is zero, connection-reset count is zero, and 5xx-free throughput dips only by the share of removed capacity. On the process side, log the shutdown timeline and verify each step: unhealthy flag set, listeners closed, idle sockets reaped, in-flight counter reaching zero or the deadline firing, exit code 0 — all within the grace period. Repeat the deploy with a deliberately slow request class to prove the deadline path completes cleanly instead of SIGKILLing.