Kubernetes Pending Pod — Schedule Failures and PVC Binding

Diagnose Kubernetes Pending pods by reading scheduler events, node allocatable resources, taints, affinity rules, and PVC binding, then correct the manifest.

beginnerdebuggingkubernetesscheduling

Symptoms

Pod stays Pending indefinitely with no container started
Events show 0/3 nodes are available: 3 Insufficient cpu
Events mention untolerated taint or node affinity mismatch
Pod has unbound immediate PersistentVolumeClaims

Possible causes

CPU or memory requests exceed what any node can allocate
Node selectors, affinity, or topology spread rules exclude every candidate
Taints on the nodes have no matching toleration in the pod
The PersistentVolumeClaim is pending because no StorageClass can provision it

Diagnosis steps

Read the scheduler's event message

kubectl describe pod -n app api-7d9c8f6b5-4xl2q | sed -n '/Events:/,$p'

Interpretation

The FailedScheduling message lists each node group's reason. Insufficient cpu means requests are too large; untolerated taint means scheduling constraints; unbound PVC means storage, not compute.

See cluster events in order

kubectl get events -n app --sort-by=.lastTimestamp | tail -30

Interpretation

Repeated FailedScheduling with the same reason confirms the scheduler cannot place the pod. A single WaitForFirstConsumer message means provisioning is waiting for scheduling, which is normal.

Compare requests with node allocatable

kubectl describe node ip-10-0-3-17 | grep -A12 -E 'Allocatable:|Allocated resources:'

Interpretation

Allocated resources already near 100 percent means the node has no room. Allocatable minus allocated is the real free capacity, not the machine's total memory.

Check PVC and quota constraints

kubectl get pvc -n app && kubectl describe resourcequota -n app

Interpretation

A PVC in Pending with no provisioner explains why the pod cannot start. An exhausted ResourceQuota rejects pod creation or leaves it unschedulable depending on the resource.

Root cause

The Kubernetes scheduler places a pod only when one node satisfies every constraint: enough allocatable CPU and memory after existing requests, matching node selector and affinity, tolerated taints, available ports if hostPort is used, and bound storage for immediate volume claims. If any constraint fails on every node, the pod remains Pending and no container is created.

Fix

Lower requests to realistic values, add nodes or enable autoscaling, and align nodeSelector and affinity with actual node labels. Add tolerations only for taints you own, and use topology spread constraints with whenUnsatisfiable: ScheduleAnyway if strict spreading is blocking placement. For storage, install a provisioner or use a StorageClass with WaitForFirstConsumer, and verify the PVC storageClassName matches.

Risk note

Reducing requests too far causes CPU throttling and OOM kills once the pod runs. Adding a broad toleration can place workloads on dedicated or unhealthy nodes. Deleting a Pending PVC may lose the claim and its data if it was already bound.

Verify the fix

Re-apply the manifest and confirm the pod transitions to Running and the event log shows a successful Scheduled message. Check kubectl get pod -o wide that the node and pod IP are assigned, and verify the PVC reports Bound.

Pending means the scheduler said no

A Pending pod has not been assigned to a node, so no container image is pulled and no application log exists. The only useful evidence is the scheduler event:

Warning  FailedScheduling  0/3 nodes are available: 3 Insufficient cpu.
Warning  FailedScheduling  0/3 nodes are available: 3 node(s) had untolerated taint {dedicated: gpu}.

Each clause after the count names a different blocker. Read all of them; fixing one may reveal another.

Check capacity, not machine size

kubectl describe node ip-10-0-3-17 | grep -A12 -E 'Allocatable:|Allocated resources:'
kubectl get pvc -n app

Requests, not actual usage, drive scheduling. A node with 16 GiB of memory but 15.5 GiB already requested has no room for another 1 GiB pod. PVCs are the other common cause: with immediate binding, the pod cannot schedule until storage exists.

Correct the constraint

Lower CPU and memory requests to measured values.
Match nodeSelector and affinity to labels that actually exist on nodes.
Add a toleration only for taints your team controls.
Use a WaitForFirstConsumer StorageClass so the volume is provisioned in the right zone.
Remove hostPort collisions and topology spread rules that no node can satisfy.

resources:
  requests:
    cpu: 100m
    memory: 128Mi
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: api

Verify

Watch the pod until it reports Running with a node name and pod IP. Confirm the events show Scheduled and the PVC is Bound. A pod that schedules but then restarts is a different problem; from here, move to container logs and probe failures.