Drain Traffic Before Maintenance and Restore It Safely

Use load balancer draining with health-check flip, watch in-flight requests complete, and restore traffic in stages with a rollback exit.

~30 mindeployoperationsnginx

Task

Take a node out of rotation for maintenance without dropping requests, then return it to service in stages.

When to use this

Database maintenance requires taking the app tier offline. The last attempt just powered nodes off, and users lost half-finished uploads mid-request.

Prerequisites

A load balancer or ingress supporting health-check-based removal
A workload with requests lasting more than a few seconds
A maintenance window and a rollback plan

Steps

1. Flip the health check instead of dropping the node.

# maintenance mode endpoint
app.get("/healthz", (req, res) => {
  if (maintenanceMode) return res.status(503).end();
  res.status(200).end();
});

Expected output: The balancer removes the node within one check interval once the endpoint reports unhealthy, without touching balancer configuration.

2. Watch in-flight requests drain.

watch -n2 'curl -s http://localhost:9090/metrics | grep -E "inflight_requests|active_connections"'

Expected output: The in-flight gauge decays to zero over the drain window, showing real requests finishing rather than being cut.

3. Confirm the balancer has stopped routing new traffic.

tail -f /var/log/nginx/access.log | grep -c " $(hostname) " || echo "no new traffic"

Expected output: Access logs on the node show no new arrivals after removal, while the rest of the fleet absorbs the traffic.

4. Perform the maintenance with a bounded window.

systemctl stop app && psql -h db.internal -c "VACUUM (ANALYZE) orders;" && systemctl start app

Expected output: Maintenance runs on a node serving zero user traffic, and the fleet above keeps serving throughout.

5. Restore in stages and verify before full weight.

curl -s http://localhost:3000/readyz && echo "ready" && # then flip health back
app.get("/healthz", (req, res) => res.status(200).end());

Expected output: The node passes its own readiness check before rejoining, and the balancer ramps traffic back over its normal health interval.

Troubleshooting

Problem

In-flight count never reaches zero

Solution

A long upload or websocket refuses to end; set a maximum drain time, warn connected clients, and only then cut, so maintenance does not wait on one stuck client forever.

Problem

Balancer keeps routing after the flip

Solution

Health interval and failure threshold are long; shorten them for the maintenance window or use the balancer API to drain explicitly.

Problem

Node rejoins before it is truly ready

Solution

Readiness must reflect warmed caches and completed migrations; gate the health flip on a readiness endpoint that only passes after warm-up.

Limitations

Draining protects requests, not sessions; stateful sessions need connection draining at the protocol level or reconnection logic in clients.
A single-node deployment cannot drain; this recipe requires at least two serving instances.

Leave politely, come back gradually

Powered-off servers cut every request in flight, and half-finished uploads are how users learn about your maintenance. The polite exit is a health-check flip: the node declares itself unhealthy, the balancer stops sending new traffic within one interval, in-flight requests finish, and maintenance happens on a quiet node. The polite return is the same door: readiness first, then health, and the balancer ramps traffic back on its own schedule.

// The traffic switch that needs no balancer access
app.get("/healthz", (req, res) => {
  if (maintenanceMode) return res.status(503).end();
  res.status(200).end();
});

Flip health, not configuration

Editing the balancer to remove a node is an operation with its own failure modes and its own rollback. Making the application report unhealthy achieves removal through machinery that runs constantly anyway, and restoration is literally returning the endpoint to 200. The maintenance procedure becomes something the app controls, from inside the app.

Verify the drain, never assume it

The drain is only real if the in-flight gauge reaches zero before you act. Watch the metric, watch the access log for new arrivals, and only proceed when both say quiet. The difference between a clean maintenance and an incident is usually the sixty seconds you waited for evidence instead of assuming.