Kubernetes — pod restarts in a loop from a failed liveness probe

Fix liveness probe restart loops by tuning thresholds and separating startup from steady-state checks.

intermediatedebuggingkubernetesprobes

Symptoms

The pod restarts repeatedly with a failed liveness probe
The app works but is killed during startup
Restart count climbs quickly
Readiness is fine but liveness keeps failing

Possible causes

initialDelaySeconds is shorter than the real startup time
The liveness endpoint depends on a slow dependency
Timeout and failure thresholds are too tight for normal load
Liveness and readiness probe the same heavy endpoint

Diagnosis steps

Read the probe failure events

kubectl describe pod <pod> | grep -A5 Liveness

Interpretation

The events show which probe failed and after how long, distinguishing a startup race from a steady-state failure.

Test the probe endpoint locally

kubectl exec <pod> -- curl -s localhost:8080/healthz

Interpretation

Manual response time tells you whether the endpoint is slow and whether timeouts are realistic.

Use a startup probe for slow boot

startupProbe: { initialDelaySeconds: 5, failureThreshold: 30 }

Interpretation

A startup probe gives boot time room while liveness stays strict for steady state.

Root cause

The liveness probe failed enough times to trigger a restart, because the application was still starting, the endpoint was slow under load, or thresholds were tighter than normal response times.

Fix

Separate startup from steady-state with a startup probe, set initialDelaySeconds and thresholds to realistic values, and make the liveness endpoint cheap and independent of slow dependencies.

Verify the fix

Deploy the change and confirm the restart count stays flat, and confirm the app becomes ready and serves traffic without being killed.

Symptom walkthrough

A pod restarts in a loop, and events attribute it to a failed liveness probe. The application may be perfectly healthy but is killed because the probe judged it dead during startup or under temporary load. Liveness is a strong signal, so a too-strict probe turns a slow start or a brief dependency slowdown into a restart loop that makes things worse.

Fix in practice

Add a startup probe for slow initialization and keep liveness strict only for the steady state. Set initialDelaySeconds, timeouts and failure thresholds from measured response times, and make the liveness endpoint cheap and not dependent on slow downstream services. Readiness should carry the dependency gating instead.

Verification

Deploy and confirm the restart count stops climbing, then confirm the app becomes ready and serves traffic without being killed. Stable restarts with working traffic confirms the probes now match real behavior.

Diagnostic commands

Check the behavior with commands like these:

kubectl describe pod <pod> | grep -A5 Liveness
kubectl exec <pod> -- curl -s localhost:8080/healthz
startupProbe: { initialDelaySeconds: 5, failureThreshold: 30 }