Add Container Health Probes That Match Reality

Separate startup, liveness, and readiness probes so slow boots are not killed and a broken dependency does not restart-loop every pod.

~30 minkuberneteshealth-checkreliability

Task

Make Kubernetes restart only genuinely wedged containers and remove unready pods from load balancing without cascading restarts.

When to use this

A service exposes one /health endpoint that checks the database. During a database failover, every pod fails liveness, Kubernetes restarts all of them, and the outage grows from a dependency blip into a full restart storm.

Prerequisites

A Kubernetes Deployment running an HTTP service
kubectl access and permission to edit the Deployment
Application endpoints that can report process and dependency state separately

Steps

1. Expose /livez for process health and /readyz for dependency and startup readiness.

app.get('/livez', (req, res) => res.status(200).send('ok'));

app.get('/readyz', async (req, res) => {
  if (!migrationsApplied) return res.status(503).send('migrating');
  const ok = await db.query('SELECT 1');
  return ok ? res.status(200).send('ready') : res.status(503).send('db down');
});

Expected output: /livez stays 200 while the process can serve, and /readyz returns 503 when migrations are pending or the database is unreachable.

2. Wire the three probe types with timeouts and thresholds suited to each signal.

startupProbe:
  httpGet:
    path: /livez
    port: 8080
  failureThreshold: 30
  periodSeconds: 2
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Expected output: Slow starts get up to 60 seconds, a wedged process restarts after about 30 seconds, and a dependency failure only removes the pod from endpoints.

3. Simulate a dependency outage and confirm the pod becomes unready without restarting.

kubectl scale deployment/payments-api --replicas=3 && kubectl get pods -w

Expected output: READY shows 0/1 during the outage while RESTARTS stays at 0.

4. Verify that a hung process is restarted by the liveness probe.

kubectl exec deploy/payments-api -- kill -STOP 1

Expected output: The liveness probe fails three times in a row and the container restarts, restoring service.

Troubleshooting

Problem

Pods enter CrashLoopBackOff during a slow first boot

Solution

Move dependency checks out of liveness, keep startupProbe generous, and ensure liveness only tests the process.

Problem

Traffic reaches a pod before migrations finish

Solution

Gate readiness on a migration flag, not merely on a successful TCP connect, and set initialDelaySeconds conservatively.

Limitations

Probes cannot detect partial failures such as a single broken query path.
Liveness restarts cannot fix a shared dependency outage and can make it worse if misconfigured.

Three probes, three questions

Startup asks whether the process has finished booting. Liveness asks whether it is still able to make progress. Readiness asks whether it should receive traffic. Conflating them is the most common probe mistake: putting a database check in liveness turns a database outage into a restart storm, and putting liveness logic in readiness keeps a wedged pod in the load balancer.

Keep liveness boring

Liveness should test only the process and its ability to serve HTTP. No database calls, no external APIs. If the Go or Node runtime is deadlocked, the endpoint will fail and a restart is the right response. If a dependency is down, the process is still healthy; readiness should remove it from traffic and let it recover.

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Make readiness meaningful

Readiness should reflect the conditions required to serve correct responses: migrations applied, connection pool initialized, and critical dependency reachable. Return 503 quickly with a short timeout so the kubelet does not hang. Fail readiness at the very start of shutdown so endpoints are removed before the listener closes.

Tune with evidence

Set timeoutSeconds below periodSeconds so a hung request does not consume the whole cycle. Use a startup probe instead of a long initialDelaySeconds; the latter wastes time on fast starts and is still too short on slow ones. After a change, watch restart counts and readiness transitions for a week. A pod that flaps between ready and unready is usually failing a dependency check that belongs in readiness, not liveness.