Guard Cluster Scale-Down from Disruption

Protect workloads from node removal with pod disruption budgets, annotate safe-to-evict, and let the autoscaler scale down only nodes it is sure are idle.

~35 minkubernetesautoscalingreliability

Task

Stop cluster scale-down from disrupting critical pods during consolidation.

When to use this

The cluster autoscaler removes a node that still runs a critical pod, and the workload has no disruption budget to prevent the eviction.

Prerequisites

A cluster autoscaler
PodDisruptionBudgets for critical workloads

Steps

1. Create PodDisruptionBudgets for workloads that must stay available.

Expected output: Evictions are limited to safe numbers.

2. Annotate pods that genuinely should not block scale-down.

Expected output: The autoscaler can reclaim their nodes safely.

3. Check that scale-down candidates have no local storage or single-replica state.

Expected output: Removal does not lose data.

4. Observe scale-down events and confirm disruptions stay within budgets.

kubectl get pdb -A

Expected output: Guardrails are verified in practice.

Troubleshooting

Problem

Scale-down blocked forever

Solution

A too-strict PDB or a pod that never becomes evictable stalls consolidation; verify the budget matches real redundancy.

Problem

Stateful pod removed

Solution

StatefulSet pods with local storage must not be evicted by consolidation; use PodDisruptionBudgets and safe annotations carefully.

Limitations

PDBs constrain voluntary disruptions but do not stop node failure or maintenance that ignores them.
Overly strict budgets waste capacity, while loose ones invite outages; balancing the two is ongoing work.

Context

Stop cluster scale-down from disrupting critical pods during consolidation. The cluster autoscaler removes a node that still runs a critical pod, and the workload has no disruption budget to prevent the eviction.

Steps

Create PodDisruptionBudgets for workloads that must stay available. Expected: Evictions are limited to safe numbers.
Annotate pods that genuinely should not block scale-down. Expected: The autoscaler can reclaim their nodes safely.
Check that scale-down candidates have no local storage or single-replica state. Expected: Removal does not lose data.
Observe scale-down events and confirm disruptions stay within budgets. Expected: Guardrails are verified in practice.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: app-pdb }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: web } }
---
# let the autoscaler evict a non-critical pod
metadata:
  annotations: { cluster-autoscaler.kubernetes.io/safe-to-evict: "true" }

Verify

Evictions are limited to safe numbers.
The autoscaler can reclaim their nodes safely.
Removal does not lose data.
Guardrails are verified in practice.

Limits

PDBs constrain voluntary disruptions but do not stop node failure or maintenance that ignores them.
Overly strict budgets waste capacity, while loose ones invite outages; balancing the two is ongoing work.