Protect Availability with a PodDisruptionBudget

Set minAvailable for critical deployments and verify that node drains respect the budget instead of evicting every replica.

~25 minkubernetesavailabilityoperations

Task

Prevent voluntary disruptions such as node upgrades and cluster autoscaling from taking a service fully offline.

When to use this

A cluster upgrade drains nodes one by one. The payments deployment has three replicas, but all three sit on the same node, so the drain evicts them together and checkout fails for four minutes.

Prerequisites

Kubernetes 1.27 or newer
A Deployment with at least three replicas
kubectl access to create policy resources

Steps

1. Create a budget that keeps two pods available during voluntary evictions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

Expected output: With three healthy pods, kubectl reports one allowed disruption.

2. Confirm the budget status and the number of currently allowed disruptions.

kubectl get pdb payments-api -n production

Expected output: MIN AVAILABLE 2, ALLOWED DISRUPTIONS 1, and CURRENT 3.

3. Drain a node and observe that eviction respects the budget.

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

Expected output: One pod is evicted, the replacement becomes Ready, and further evictions wait until capacity returns.

4. Alert when allowed disruptions drop to zero for longer than a maintenance window.

kube_poddisruptionbudget_status_pods_healthy{namespace="production"}
  - kube_poddisruptionbudget_status_desired_healthy{namespace="production"} < 0

Expected output: A budget that is blocking drains pages the owner before the next maintenance window starts.

Troubleshooting

Problem

The drain hangs indefinitely

Solution

Allowed disruptions is zero because fewer than minAvailable pods are healthy; scale up or use maxUnavailable with a percentage that tolerates one pod.

Problem

The PDB matches no pods and reports zero disruptions allowed

Solution

Check the selector labels against the pod template; a PDB whose selector matches nothing blocks every drain and protects nothing.

Limitations

A PDB only governs voluntary disruptions; node crashes and kernel failures can still take all replicas at once.
minAvailable equal to replica count blocks all drains; use N-1 or a maxUnavailable percentage instead.

Voluntary disruption is controllable

Node drains, cluster upgrades, and autoscaler scale-down are voluntary disruptions: the cluster asks pods to leave. A PodDisruptionBudget tells the eviction API how many pods may be unavailable at once. It does not make the workload highly available by itself; it prevents maintenance operations from removing too many replicas simultaneously.

Choose the arithmetic

minAvailable: 2 with three replicas allows one eviction at a time. maxUnavailable: 1 is equivalent here but scales with replica count if you later increase it. Prefer percentages for large fleets. Never set minAvailable equal to replicas: that makes every drain wait forever. Also spread pods across nodes with topology spread constraints, because a budget cannot help if all replicas live on the drained node.

spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api

Test the drain

Run kubectl get pdb and confirm ALLOWED DISRUPTIONS is positive. Drain a node and watch pods move one at a time. If the drain blocks, inspect healthy versus desired pods and check whether a second PDB selects the same pods. Overlapping budgets combine conservatively and can deadlock maintenance.

Monitor the guardrail

A budget that is permanently at zero allowed disruptions is an outage waiting for the next upgrade. Alert on it. Review budgets after changing replica counts or autoscaling policies, and keep them in version control with the Deployment. Remember that a PDB protects against eviction, not against a bad rollout; readiness probes and maxSurge still govern rolling updates.