Debug a CrashLoopBackOff in Ten Minutes

Read previous container logs, exit codes, and event messages to classify the crash cause, then fix the right layer instead of restarting harder.

~20 mindeploykubernetesdebugging

Task

Extract the previous container logs and exit code, classify the failure, and apply the fix at the correct layer: command, config, or probe.

When to use this

A freshly deployed pod enters CrashLoopBackOff. The team restarts the deployment twice, and the only visible error is a generic restart counter.

Prerequisites

kubectl access to the cluster and namespace
A pod in CrashLoopBackOff or the ability to reproduce one
Container image debugging basics (sh, env, ls)

Steps

1. Read the previous, not current, container logs.

kubectl get pods -l app=worker | head -3; kubectl logs worker-7d9c6b5-x2lkp --previous --tail=50

Expected output: The last lines before the crash print, typically a stack trace, a missing file, or a config parse error naming the failing component.

2. Get the exit code and signal from the pod status.

kubectl get pod worker-7d9c6b5-x2lkp -o jsonpath="{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}"

Expected output: A line like worker exit=1 reason=Error or exit=137 reason=OOMKilled separates application crash from memory kill in one glance.

3. Inspect events for scheduling and probe problems.

kubectl describe pod worker-7d9c6b5-x2lkp | sed -n '/Events:/,$p' | tail -12

Expected output: Events reveal causes like Unhealthy readiness probe, failed ConfigMap mount, or FailedScheduling, each pointing to a different fix.

4. Reproduce inside the image to confirm the failing layer.

kubectl run debug --rm -it --image=registry.example.com/worker:1.4.2 --restart=Never -- sh -c "env | grep -i config; ls /app/config || echo MISSING"

Expected output: The interactive run shows whether the expected env var and config file exist, confirming or eliminating config as the cause before editing manifests.

5. Apply the layer-specific fix and watch the rollout.

kubectl set env deployment/worker CONFIG_PATH=/app/config/prod.yaml && kubectl rollout status deployment/worker --timeout=120s

Expected output: The rollout completes with the pod Ready, and restart count stays at zero instead of climbing every few minutes.

Troubleshooting

Problem

Logs are empty or truncated

Solution

The app writes to a file instead of stdout; check kubectl exec -- ls /var/log/app, or set the container command to tail the file until logging is fixed.

Problem

Exit code 137 immediately on start

Solution

That is OOMKilled: raise memory limits after confirming a real need with a load test, and grep the app for heap settings that exceed the container limit.

Problem

Pod runs fine interactively but crashes in deployment

Solution

A liveness probe is killing it before startup finishes; add a startupProbe or lower initialDelaySeconds instead of loosening every check.

Limitations

kubectl logs --previous holds only the last terminated container; earlier crashes need log shipping or events.
CrashLoopBackOff backoff timing is exponential, so verification after a fix can take minutes; watch rollout status rather than polling restarts.

The previous container is the one that crashed

CrashLoopBackOff hides its own evidence: the current container has just started and logged nothing useful. The logs of the previous, terminated container, plus its exit code and the events around it, classify the failure into application error, OOM kill, or probe kill within minutes.

# What did the LAST run say before dying?
kubectl logs worker-7d9c6b5-x2lkp --previous --tail=20
# Error: ENOENT: no such file or directory, open '/app/config/prod.yaml'

# How did it die?
kubectl get pod worker-7d9c6b5-x2lkp -o jsonpath=\
  "{.status.containerStatuses[0].lastState.terminated.reason}"
# Error   (not OOMKilled -> app-level failure)

# Fix the named layer, then verify once
kubectl set env deployment/worker CONFIG_PATH=/app/config/prod.yaml
kubectl rollout status deployment/worker --timeout=120s

Exit code is the first triage question

exit=1 with reason=Error is the application itself failing: read the previous logs. exit=137 with reason=OOMKilled is the kernel killing the container for exceeding memory limits: raise or rightsize the limit. exit=0 in a loop usually means the process finished and exited, and the container has no supervisor: fix the command or make it wait.

Do not restart harder

Restarting the deployment again resets nothing but the backoff timer, and each crash burns the same broken state. Reproduce interactively with the same image to see exactly what the process sees, fix the layer the evidence names, and let one rollout verify it. Restart count staying at zero is the only honest success metric.