Isolate Workloads with Taints and Tolerations

Taint special nodes so only workloads that tolerate them are scheduled there, and pair it with node affinity so pods land where they belong.

~30 minkubernetesschedulinginfrastructure

Task

Reserve a class of nodes for specific workloads using taints and tolerations.

When to use this

GPU or memory-heavy nodes get filled by ordinary pods, so the workloads meant for them cannot schedule and sit pending.

Prerequisites

A cluster with differentiated node pools
Access to set taints on nodes

Steps

1. Taint the reserved nodes with a dedicated key and effect.

Expected output: Ordinary pods are repelled from those nodes.

2. Add matching tolerations to only the workloads that should run there.

Expected output: Intended workloads can schedule on the nodes.

3. Add node affinity or selectors for specificity.

Expected output: Toleration permits, affinity directs.

4. Verify with scheduling that other pods stay off the tainted nodes.

kubectl taint nodes gpu-1 dedicated=gpu:NoSchedule && kubectl describe node gpu-1 | grep -i taint

Expected output: Isolation holds.

Troubleshooting

Problem

All pods avoid a node

Solution

A toleration permits but does not attract; check affinity and whether the taint is more restrictive than intended.

Problem

Tolerations too broad

Solution

A toleration with no value or key matches many taints; scope it precisely.

Limitations

Taints depend on node labels being set correctly and consistently, which needs automation.
They do not guarantee resource fit or priority; pair with affinity and QoS classes.

Context

Reserve a class of nodes for specific workloads using taints and tolerations. GPU or memory-heavy nodes get filled by ordinary pods, so the workloads meant for them cannot schedule and sit pending.

Steps

Taint the reserved nodes with a dedicated key and effect. Expected: Ordinary pods are repelled from those nodes.
Add matching tolerations to only the workloads that should run there. Expected: Intended workloads can schedule on the nodes.
Add node affinity or selectors for specificity. Expected: Toleration permits, affinity directs.
Verify with scheduling that other pods stay off the tainted nodes. Expected: Isolation holds.

tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
nodeSelector:
  node-pool: gpu

Verify

Ordinary pods are repelled from those nodes.
Intended workloads can schedule on the nodes.
Toleration permits, affinity directs.
Isolation holds.

Limits

Taints depend on node labels being set correctly and consistently, which needs automation.
They do not guarantee resource fit or priority; pair with affinity and QoS classes.