Graceful Shutdown Design

Improve service reliability: design graceful shutdown, tune health probes, and verify the system degrades gracefully under partial failure.

intermediate~30 minreliabilityresilienceoperations

Objective

Make the service survive restarts and partial dependency failures without dropping in-flight requests.

Environment

A working reliability environment with access to the target system and its tooling
Configuration and credentials for the reliability workflow installed and verified
A rollback or undo path documented before any change is applied

Steps

1. Implement graceful shutdown

On SIGTERM, stop accepting new requests, drain in-flight work, and exit only after the drain completes or the grace period ends.

Expected state: A rolling restart drops zero in-flight requests.

2. Tune health probes

Set the readiness probe to fail fast on dependency loss and the liveness probe to avoid killing the pod during a slow but healthy startup.

Expected state: Probes correctly reflect readiness and do not cause false restarts.

3. Verify graceful degradation

Inject a dependency failure and confirm the service returns a controlled fallback instead of a 5xx cascade.

Expected state: The service serves a fallback response and recovers when the dependency returns.

Decision points

How long should the drain window be?

Longer than the p99 request duration
A fixed 30 seconds

Size the drain window to the p99 request duration plus a margin; a fixed value may cut off slow requests.

Expected outcome

Make the service survive restarts and partial dependency failures without dropping in-flight requests. The change is verified and a rollback path is confirmed.

Verification

Make the service survive restarts and partial dependency failures without dropping in-flight requests. Compare before and after metrics and confirm the expected state is reached.

Cleanup

Archive the run artifacts, remove temporary credentials, and update the runbook with any changes.

Overview

Improve service reliability: design graceful shutdown, tune health probes, and verify the system degrades gracefully under partial failure. The goal is to make the service survive restarts and partial dependency failures without dropping in-flight requests.

kubectl rollout restart deployment/api -n prod

Steps

Implement graceful shutdown. On SIGTERM, stop accepting new requests, drain in-flight work, and exit only after the drain completes or the grace period ends. Expected: A rolling restart drops zero in-flight requests.
Tune health probes. Set the readiness probe to fail fast on dependency loss and the liveness probe to avoid killing the pod during a slow but healthy startup. Expected: Probes correctly reflect readiness and do not cause false restarts.
Verify graceful degradation. Inject a dependency failure and confirm the service returns a controlled fallback instead of a 5xx cascade. Expected: The service serves a fallback response and recovers when the dependency returns.

Verification

Confirm the outcome by checking: make the service survive restarts and partial dependency failures without dropping in-flight requests. Run the verification command and compare the result against the expected state. If any check fails, stop and investigate before continuing.

Decision points

How long should the drain window be?
- Longer than the p99 request duration
- A fixed 30 seconds
- Recommendation: Size the drain window to the p99 request duration plus a margin; a fixed value may cut off slow requests.