Stop Slow-Starting Containers from Being Killed

Give JVM or migration-heavy containers a startupProbe with a generous budget so liveness never executes during boot and restart loops end.

~20 mindeploykubernetesoperations

Task

Configure startup, liveness, and readiness probes with correct budgets so a container that needs ninety seconds to boot is never restarted for it.

When to use this

A Java service takes eighty seconds to start. The default ten-second liveness probe kills it at fifteen seconds, it restarts, and the pod enters CrashLoopBackOff forever.

Prerequisites

A Kubernetes cluster running a slow-booting workload
Knowledge of the realistic startup duration from logs
Permission to patch the pod spec

Steps

1. Measure the true startup duration.

kubectl logs pod/app-7d9c --previous | grep -iE "started|ready" | head -3

Expected output: Logs show the container becoming ready after about eighty seconds, far beyond any default probe budget.

2. Add a startup probe sized to the real duration.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30   # 5s * 30 = 150s budget

Expected output: Liveness and readiness are suppressed until the startup probe passes, so the slow boot is expected rather than punished.

3. Keep liveness strict after startup.

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Expected output: Once serving, a genuinely hung process is restarted within thirty seconds, preserving the safety liveness provides.

4. Separate readiness from aliveness.

readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

Expected output: A pod that is alive but not yet able to serve is removed from endpoints instead of being restarted.

5. Verify the pod reaches Running without restarts.

kubectl rollout status deploy/app && kubectl get pods -l app=app -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount

Expected output: The pod starts once, restarts stay at zero, and the rollout completes on the first attempt.

Troubleshooting

Problem

Startup budget exceeded and pod restarts anyway

Solution

The app genuinely hangs during boot, for example waiting on a mandatory dependency; fail fast in the app or add an initContainer that gates on the dependency instead of stretching the probe.

Problem

Liveness still kills during long requests

Solution

A request-handler blocked for thirty seconds fails a liveness probe implemented on the same thread; check liveness on a separate path served by a dedicated handler.

Problem

Probe path returns 404 during early boot

Solution

The route registers late; have the handler bind and return 200 immediately, or probe a tcpSocket until the health route exists.

Limitations

A generous startup budget hides real regressions in boot time; track startup duration as a metric so drift stays visible.
Probes cannot distinguish slow from stuck by themselves; the application must make health endpoints honest.

Slow is not broken

Liveness probes exist to restart hung processes, but they cannot tell hung from booting. A JVM or an app running migrations needs a minute before it can answer anything, and a default liveness budget converts that into an infinite restart loop. The startup probe fixes the semantics: while it has not passed, nothing else runs; once it passes, liveness enforces its strict budget against a process that should genuinely be responsive.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30   # 150-second boot budget

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Three probes, three meanings

Startup means the process is still initializing; liveness means it should be able to answer and must be restarted if not; readiness means it may receive traffic now. Conflating them produces the classic failure of restarting a perfectly healthy pod that simply has an exhausted connection pool, when the right response was to stop routing to it for ten seconds.

Budget from measurements, not vibes

The startup budget should be the p99 boot duration with generous headroom, read from logs or metrics, not a guess. Then monitor actual startup durations: when the budget was sized for ninety seconds and boot drifts to one hundred forty, the probe starts killing again, and the metric tells you weeks before the pager does.