阅读中文版本

Kubernetes Pod Pending — Taint Not Tolerated

Fix unschedulable pods by adding a matching toleration, removing the taint, or adjusting node affinity and node selectors.

kubectl describe pod <pod> | grep -A 5 "Events"

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Symptoms

The error “Kubernetes Pod Pending — Taint Not Tolerated” points at a specific failure mode. This guide documents the symptoms you can observe, the commands that isolate the cause, the root cause, the fix, and the checks that prove the fix works.

Observed symptoms include:

• Pod stays Pending with “has taints that pod does not tolerate”
• All nodes are tainted and no pod can schedule
• Scheduling works for one workload but not another

Reproduce and Diagnose

Reproduce the failure in isolation before changing any code. Work through the checks below in order; each one narrows the search space.

Step 1: Read the scheduler message

kubectl describe pod <pod> | grep -A 5 "Events"

The FailedScheduling event lists the taints on candidate nodes and the missing toleration.

Step 2: List node taints and labels

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Seeing the taints tells you exactly which toleration to add.

Likely contributors, in order of frequency:

• The target nodes carry a taint (for example dedicated=gpu:NoSchedule) the pod does not tolerate
• A nodeSelector or affinity restricts placement to nodes that do not exist
• The control-plane node is tainted and the workload expects it

Root Cause

A taint repels pods unless they declare a matching toleration. With NoSchedule, the scheduler refuses to place untolerated pods on tainted nodes, and if no untainted node fits, the pod stays Pending.

The Fix

Add a toleration matching the node taint (with the correct effect), or remove the taint if it is no longer intended, and reconcile node selectors and affinity with existing node labels.

Watch out: Tolerating a taint does not guarantee placement on that node; affinity rules must also select it if dedicated placement is intended.

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule
  affinity:
    nodeAffinity:   # also select the dedicated nodes
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: dedicated
                operator: In
                values: ["gpu"]

Verify and Prevent

Confirm the pod schedules onto a node and reaches Running; verify the FailedScheduling event no longer appears.

Once verified, add a regression test that reproduces the original symptom so the failure cannot silently return.

This class of failure is usually environmental rather than pure logic, so capture the exact command, the environment variables, and the working directory in your bug report. Reproduce it in the smallest possible script, then widen the scope only once the minimal case is understood. A short written checklist for this failure mode saves hours the next time it appears.