Drain a Node Without Dropping Requests
Cordon first, drain with explicit flags, respect disruption budgets, and verify workloads moved before terminating the node.
Task
Retire or patch a node while keeping every service available and knowing exactly why a drain is stuck.
When to use this
An engineer drains a node with force flags, evicting every pod at once and deleting local cache data that a batch job still needs.
Prerequisites
- kubectl access with permission to patch nodes
- PodDisruptionBudgets for critical workloads
- Workloads that can be rescheduled onto other nodes
Steps
1. Cordon the node so no new pods are scheduled onto it.
kubectl cordon ip-10-0-3-17Expected output: The node shows SchedulingDisabled and new pods are placed elsewhere.
2. List what will be evicted, including stateful pods that need special handling.
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-3-17 -o wideExpected output: A complete list of pods on the node, including DaemonSets that drain will ignore.
3. Drain with a timeout and explicit handling for DaemonSets and emptyDir data.
kubectl drain ip-10-0-3-17 --ignore-daemonsets --delete-emptydir-data --timeout=300sExpected output: Pods are evicted one by one and the command returns 0 when the node is empty.
4. Confirm that only DaemonSet pods remain and replacements are healthy.
kubectl get pods -A -o wide --field-selector spec.nodeName=ip-10-0-3-17 && kubectl get pods -A -o wide | grep -v ip-10-0-3-17 | grep paymentsExpected output: Only DaemonSet pods stay on the node, and evicted workloads are Running and Ready on other nodes.
Troubleshooting
Problem
Drain hangs and reports a disruption budget violation
Solution
The budget allows no more evictions. Scale the workload up or temporarily adjust the budget after verifying spare capacity.
Problem
Local data disappeared during the drain
Solution
emptyDir data is deleted on eviction. Back it up or move the workload to a StatefulSet with persistent volumes.
Limitations
- Draining does not migrate pods; the scheduler places them, and a full cluster leaves replacements Pending.
- DaemonSet pods remain until the node is deleted, which is expected behavior.
Drain is eviction with rules
kubectl drain does not move pods; it cordons the node and asks the eviction API to remove them, one at a time. The eviction API honors PodDisruptionBudgets, which is the whole point: a budget that requires two available replicas will refuse an eviction that would leave one. When every pod is gone, the node is safe to reboot or terminate. The two flags people reach for in a hurry, --force and --delete-emptydir-data, bypass parts of that safety, so use them deliberately rather than by habit.
Inspect before you evict
List the pods on the node first. DaemonSets are expected and will stay; a StatefulSet pod with a local volume may need a manual step; a batch job with an emptyDir may hold the only copy of its intermediate output. Cordon before draining so the node does not refill while you work. Then drain with a timeout, because a drain that hangs forever is worse than one that fails and tells you which budget is blocking it.
kubectl cordon ip-10-0-3-17
kubectl drain ip-10-0-3-17 --ignore-daemonsets --delete-emptydir-data --timeout=300s
kubectl get pods -A --field-selector spec.nodeName=ip-10-0-3-17 -o wideVerify the replacement, not the command
A successful drain means the node is empty, not that the service is healthy. Check that evicted pods reached Running and Ready elsewhere, and watch the service error rate during the operation. If replacements stay Pending, the cluster lacks capacity and you have moved the incident rather than avoided it. Uncordon the node when maintenance finishes, and record the drain in the maintenance log so the next person knows why the node was empty.