Kubernetes readiness probe failing — Running but Never Ready

Diagnose Kubernetes readiness probe failures by testing the probe endpoint inside the pod, reviewing probe timing and app dependencies, then fixing startup and health routes.

beginnerdebuggingkuberneteshealth-checks

Symptoms

Pod status is Running but READY shows 0/1
Events repeat Readiness probe failed: HTTP probe failed with statuscode: 503
Service has no endpoints and traffic returns connection errors
The container works when tested manually but never becomes Ready

Possible causes

The probe path or port does not match the application's health endpoint
initialDelaySeconds or startupProbe timing is shorter than application startup
The health endpoint checks a dependency that is not reachable from the pod
The application binds to 127.0.0.1, so the kubelet cannot reach it on the pod IP

Diagnosis steps

Confirm pod state and readiness

kubectl get pod -n app -l app=orders -o wide && kubectl get endpointslices -n app -l kubernetes.io/service-name=orders

Interpretation

A pod with READY 0/1 is excluded from Service endpoints, so traffic fails even though the process runs. Empty endpoints confirm the readiness gate.

Read the probe failure detail

kubectl describe pod -n app orders-6b7d9c5f8-hzq4m | grep -A12 -E 'Readiness:|Liveness:|Startup:'

Interpretation

The event states the HTTP status or connection error and the port used. Compare that port with the containerPort and the actual listener.

Run the probe from inside the container

kubectl exec -n app orders-6b7d9c5f8-hzq4m -- wget -qO- http://127.0.0.1:8080/healthz; echo exit=$?

Interpretation

A success here with a failing probe suggests a bind address or timing problem. A failure here confirms the endpoint itself is broken or slow.

Check startup logs and dependency errors

kubectl logs -n app orders-6b7d9c5f8-hzq4m --tail=100

Interpretation

Migrations, missing environment variables, or connection refused to a database explain why readiness never turns true. Look for the health endpoint returning 503 by design.

Root cause

Kubelet marks a container Ready only when its readiness probe succeeds. The probe is an HTTP, TCP, or exec check executed from the node against the pod IP, not from inside the container's loopback. A health route that is missing, returns 503 because a dependency is down, or starts listening after the probe deadline keeps the pod out of Service endpoints indefinitely.

Fix

Point the readiness probe at a dedicated endpoint that reports whether this instance can serve traffic, and set initialDelaySeconds, periodSeconds, timeoutSeconds, and failureThreshold to match real startup and latency. Use a startupProbe for slow boots so liveness does not restart the container. Make the app listen on 0.0.0.0, and keep readiness checks shallow enough that a transient dependency blip does not remove every replica.

Risk note

A readiness probe that checks the database can take the whole service out during a brief database failover. Setting failureThreshold too low causes flapping; too high keeps dead pods in rotation. Readiness failure does not restart the container, so it can hide a crash loop elsewhere.

Verify the fix

Confirm kubectl get pod shows READY 1/1 and the EndpointSlice lists the pod IP and port. Send a request through the Service and verify it reaches the pod, then restart the pod and check it becomes Ready only after the application is actually serving.

Running is not Ready

Running only means the container process started. Ready means the readiness probe passed and the pod was added to Service endpoints. A 0/1 pod is intentionally removed from load balancing, which is why users see connection failures while the pod looks alive.

Warning  Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 503

Test the exact probe target

Kubelet connects to the pod IP on the declared port, not to localhost inside the container:

kubectl get pod -n app -l app=orders -o wide
kubectl describe pod -n app orders-6b7d9c5f8-hzq4m | grep -A10 'Readiness:'
kubectl exec -n app orders-6b7d9c5f8-hzq4m -- wget -qO- http://127.0.0.1:8080/healthz
kubectl logs -n app orders-6b7d9c5f8-hzq4m --tail=100

If the in-container request returns 503, the app is telling the truth: a dependency is unavailable or startup is incomplete. If it returns 200 but the probe fails, check the bind address (0.0.0.0), the declared port, and probe timing.

Give startup its own budget

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 24
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

Use a startup probe for slow boots, and keep the readiness endpoint focused on whether this replica can serve traffic. A readiness check that hard-fails on a shared dependency will remove every replica during a brief outage.

Verify

Wait for READY 1/1, then confirm the EndpointSlice contains the pod IP and port. Curl the Service and check the response comes from the pod. Restart the pod once more and watch the transition: it should become Ready only after the app answers the probe, and should leave endpoints within seconds when the app is stopped.