Reload nginx Without Dropping Connections

Use nginx -s reload with correct worker shutdown settings, verify old workers drain, and confirm zero 502s during configuration changes.

~20 mindeploynginxoperations

Task

Apply nginx configuration changes with a graceful reload and prove that in-flight requests complete without errors.

When to use this

Every TLS certificate renewal or routing change causes a burst of 502s. The team schedules config changes for Sunday nights to hide the outage.

Prerequisites

nginx 1.18+ with a workload that has continuous in-flight requests
Access to run nginx -s reload and read worker PIDs
A load generator to keep requests flowing during the test

Steps

1. Record worker PIDs before the reload.

pgrep -f "nginx: worker" | sort | tee /tmp/workers-before.txt

Expected output: The current worker process ids are saved as the baseline for comparison.

2. Send the reload signal while traffic flows.

hey -n 100000 -c 50 -m GET https://api.example.com/healthz & sleep 2; nginx -s reload; wait

Expected output: The load test completes with zero non-200 responses while the reload happens underneath it.

3. Verify old workers drained and exited.

sleep 5; pgrep -f "nginx: worker" | sort | tee /tmp/workers-after.txt; comm -12 /tmp/workers-before.txt /tmp/workers-after.txt | wc -l

Expected output: Old PIDs are gone and new ones appeared; zero overlap means shutdown completed within seconds.

4. Tune worker shutdown for slow clients.

worker_shutdown_timeout 10s;

# keepalive between nginx and upstream
upstream app {
    server 127.0.0.1:3000;
    keepalive 32;
}

Expected output: Workers stop accepting new connections, finish serving current requests, and are force-closed after ten seconds at the latest.

5. Confirm the config actually took effect.

nginx -t && nginx -T 2>/dev/null | grep -A3 "server api.example.com" | head -8

Expected output: The rendered configuration shows the new settings, proving the reload applied the intended change.

Troubleshooting

Problem

502s still appear during reload

Solution

The upstream drops connections when nginx reloads; enable keepalive on the upstream block and verify the app drains connections gracefully on its side.

Problem

Old workers linger for minutes

Solution

A client holds a long request; set worker_shutdown_timeout so stuck connections cannot block config rollouts indefinitely.

Problem

Reload returns OK but nothing changes

Solution

The config failed validation and nginx kept the old one; always run nginx -t before and after, and read the error log for the actual reason.

Limitations

Reload swaps workers but cannot apply changes that require a binary upgrade, such as a new nginx version.
Long-lived connections like WebSockets may be cut by worker_shutdown_timeout; coordinate it with client reconnect logic.

A reload is a handover, not a restart

nginx -s reload starts new workers with the fresh configuration and tells old workers to finish what they are holding and exit. Done correctly, in-flight requests complete untouched and the change is invisible to clients. The failure modes are specific and fixable: upstream connections torn down by the app, or ancient workers surviving forever on one stuck download. Both are one directive away from correct.

# The graceful handover
pgrep -f "nginx: worker" > /tmp/before
nginx -t && nginx -s reload
sleep 5
pgrep -f "nginx: worker" > /tmp/after
comm -12 /tmp/before /tmp/after | wc -l   # 0 = old workers drained

Prove it with load, not with hope

The reload claims are testable in two minutes: start a load generator, reload under it, and count non-200 responses. Zero failures under fifty concurrent connections during the swap is the acceptance test, and it turns Sunday-night config changes into any-time config changes.

Bound the drain, always

Without worker_shutdown_timeout, one slow client can keep an old worker, and its old configuration, alive for hours. Ten seconds is usually right: enough for normal requests to complete, short enough that a stuck connection cannot delay the next rollout. The same number belongs in your connection-draining story on the load balancer above nginx.