Graceful Shutdown

Graceful shutdown stops accepting new work, drains in-flight requests, and closes resources before the process exits, avoiding dropped responses and corrupted state.

operations

Definition

Graceful shutdown is the process of transitioning a running service from accepting traffic to fully stopped without abruptly terminating in-flight work. It typically begins with a termination signal, stops new connections or job claims, waits for current requests or tasks to finish within a deadline, closes database and message-broker connections, and then exits. A forced exit follows if the deadline is exceeded.

Context and scope

Rolling deployments, autoscaling, and node drains terminate processes routinely. Without graceful shutdown, clients see connection resets, writes are cut off mid-transaction, and messages can be lost or processed twice. The pattern is required for zero-downtime deployment and reliable queue consumers.

Common misconceptions

Calling process.exit() immediately is fine because the load balancer will retry — retries may be unsafe and in-flight work is lost
Graceful shutdown only applies to HTTP servers — workers, schedulers, and stream consumers also need to drain
A long drain timeout is always safer — it delays deployments and can exceed the platform's termination grace period

Anti-patterns

Handling SIGTERM by exiting immediately while requests are still active
Draining without stopping new work first, so the queue never empties
Ignoring the platform's grace period and being killed before cleanup completes

Related concepts

Related examples

Authoritative source

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

The shutdown sequence

A safe sequence is: receive the signal, fail readiness checks or stop the listener, wait for in-flight requests, close resources, then exit. Stopping new work first is essential; otherwise the drain never completes because new requests keep arriving.

const server = app.listen(3000);

async function shutdown(signal) {
  console.log(`${signal} received, draining`);
  server.close(); // stop accepting new connections
  await drainInFlight({ timeoutMs: 20_000 });
  await database.end();
  process.exit(0);
}

process.on('SIGTERM', () => void shutdown('SIGTERM'));
process.on('SIGINT', () => void shutdown('SIGINT'));

server.close() stops new connections but does not itself wait for all responses in older Node.js versions; track active requests or use a library that exposes drain completion.

Deadlines and platform limits

Kubernetes sends SIGTERM, then kills the pod after terminationGracePeriodSeconds. Your drain timeout must be shorter than that period, leaving time for cleanup. Load balancers and service meshes also need time to stop routing to the instance; readiness gating is often more reliable than relying on signal timing alone.

When to use it

Use graceful shutdown for every long-running service: HTTP APIs, queue consumers, stream processors, and scheduled workers. Compared to a crash-only design, graceful shutdown protects correctness at the cost of a short delay. Compared to checkpointing, it is simpler but only covers work that can finish within the deadline.

Pitfalls

A drain that waits forever blocks deployment. A drain that stops the listener but forgets background jobs still loses work. A worker that acknowledges a message before processing it cannot recover from a mid-shutdown failure. Make shutdown idempotent and test it by sending SIGTERM during a slow request.

Graceful shutdown draining in-flight work