Connection Draining: Retiring an Instance Without Dropping Requests

Connection draining removes an instance from rotation and waits for in-flight work to finish, turning a hard cutover into an invisible handover for clients.

operations

Definition

Connection draining is the shutdown sequence that marks an instance draining, stops new traffic from being routed to it, and waits a bounded period for existing requests and connections to complete before termination.

Context and scope

Every rolling deploy, scale-in event, and node upgrade retires instances; without draining, users see resets and 502 responses during routine operations.

Common misconceptions

Draining means waiting for the process to exit — it means waiting for traffic to stop arriving
A long drain timeout is safer — it delays every deploy and hides leaks in request handling
Keep-alive connections end on their own — idle pooled connections can hold a backend in rotation indefinitely

Anti-patterns

Sending SIGTERM and closing listeners in the same handler
Setting the load balancer drain timeout shorter than the application shutdown grace period
Draining without deregistering from service discovery

Related concepts

Authoritative source

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/connection-draining.html

Two phases, one handover

Draining has two distinct phases. First, the instance is removed from routing: the load balancer stops sending new requests and the registry marks the endpoint as leaving. Second, the process keeps serving until in-flight requests finish or a deadline expires. Skipping either phase produces dropped connections.

Why shutdown order matters

A common failure is closing the listener and then sleeping. The sleep does nothing, because new connections are already refused while the load balancer still believes the instance is healthy. The correct order is: fail readiness, wait for the load balancer to notice, then stop accepting, then finish in-flight work, then exit.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 15"]
terminationGracePeriodSeconds: 60

The preStop pause covers the window between deregistration and the last routed request; the grace period must exceed the slowest legitimate request.

Sizing the window

Set the drain window from measured request duration, not from intuition. If p99 is 1.5 seconds and a batch endpoint can run for 40 seconds, a 5-second drain will cut batch work. Track connections closed during shutdown and alert when they are non-zero.

Testing the handover

Draining fails silently unless you test it. During a staging deploy, hold a long request open and confirm it completes while new requests land elsewhere. Check that the load balancer logs show the endpoint leaving before the process exits, and that no 502 responses appear at the moment of termination. These checks are cheap and catch ordering mistakes that only appear under load.

Compared to hard cutover

A hard cutover is simpler and works for stateless, short-request services behind a fast retry layer. Draining costs deploy time but protects long-lived connections, streaming responses, and clients without retries. Most platforms should drain by default and shorten the window only with evidence.