Drain Connections Before Pod Termination

Combine a preStop sleep, a long enough grace period, and readiness gating so in-flight requests finish before Kubernetes removes a pod from service.

~30 minkubernetesdeploymentreliability

Task

Prevent 502 responses during rolling deploys by letting load balancers notice a terminating pod before the process stops accepting requests.

When to use this

A checkout deployment gets a burst of 502s on every rollout. Kubernetes sends SIGTERM and removes the endpoint at the same time, but ingress controllers and kube-proxy need a moment to update rules, so requests still arrive after the server has closed its listener.

Prerequisites

A Kubernetes Deployment with a readiness probe
A service that handles SIGTERM and shuts down gracefully
kubectl access and a load-testing tool such as hey

Steps

1. Add a preStop sleep and a grace period that exceeds the sleep plus the application drain time.

spec:
  terminationGracePeriodSeconds: 40
  containers:
    - name: payments-api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 10"]

Expected output: The container stays alive for 10 seconds after termination starts while endpoints are being updated.

2. Stop accepting new connections on SIGTERM and give in-flight requests time to complete.

srv := &http.Server{Addr: ":8080", Handler: router}
go func() {
  if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
    log.Fatal(err)
  }
}()

stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM)
<-stop

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
  log.Printf("shutdown: %v", err)
}

Expected output: New connections are refused after SIGTERM while active requests complete within 25 seconds.

3. Watch endpoints during a rollout to confirm removal happens before the process exits.

kubectl get endpointslices -l kubernetes.io/service-name=payments-api -w

Expected output: The terminating pod address is removed from ready endpoints while the pod is still Running.

4. Run constant traffic through a rollout and check for failed requests.

hey -z 60s -c 20 https://api.example.com/health

Expected output: Zero non-2xx responses during kubectl rollout restart deployment/payments-api.

Troubleshooting

Problem

A short burst of 502s still appears at the start of the rollout

Solution

Increase the preStop sleep to cover ingress and kube-proxy propagation, typically 5 to 15 seconds, and make sure the readiness probe fails immediately on SIGTERM.

Problem

The process is SIGKILLed before draining completes

Solution

Raise terminationGracePeriodSeconds above preStop sleep plus the server shutdown timeout; the default 30 seconds is often too small.

Limitations

The preStop sleep is wall-clock time and lengthens every deployment.
Clients holding long-lived connections can outlast the grace period and must reconnect.

The race at termination

When a pod is deleted, two things happen in parallel: the API server marks it Terminating and the kubelet sends SIGTERM. Endpoint removal is asynchronous. Ingress controllers, kube-proxy, and cloud load balancers each watch endpoints on their own schedule, so traffic can keep arriving for several seconds after your process has stopped listening. That gap produces connection refused errors and 502 responses.

Add a drain window

The preStop hook runs before SIGTERM. Sleeping for 10 seconds keeps the process alive while endpoint updates propagate. After that the application receives SIGTERM, stops accepting new connections, and waits for active requests. The grace period must cover the preStop sleep plus the shutdown timeout, plus a small buffer; otherwise the kubelet sends SIGKILL mid-drain.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]
terminationGracePeriodSeconds: 40

Verify with traffic

Run a load generator during kubectl rollout restart and watch the non-2xx count. In parallel, stream EndpointSlices and confirm the terminating address disappears from ready endpoints before the container exits. If 502s persist, check whether the ingress controller has its own endpoint sync interval or whether a cloud load balancer health check is slower than the preStop window.

Tune, do not guess

Measure the delay between deletionTimestamp and the last request reaching the pod. Set preStop slightly above that value. Keep the readiness probe cheap and make it fail once shutdown begins, so even controllers that re-check readiness stop routing. Sidecars complicate ordering; native sidecar containers or a shared termination signal are safer than independent shutdown timers.