Review Cluster Capacity Headroom

Compare allocatable CPU and memory with requests, simulate losing one node, and record the headroom needed before peak season.

~35 minkubernetescapacityreliability

Task

Know whether the cluster survives a node failure or a traffic spike without emergency scaling at the worst moment.

When to use this

The cluster runs at 85 percent memory requests. A node fails during a promotion, replacement pods stay Pending, and the autoscaler cannot add nodes because the node group is at its maximum size.

Prerequisites

Kubernetes with kubectl access
metrics-server or a capacity dashboard
Cluster autoscaler configuration or node group limits

Steps

1. Record allocatable CPU and memory for every node, not the raw instance size.

kubectl get nodes -o custom-columns='NODE:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory'

Expected output: A per-node table of allocatable resources after kube-reserved and system reservations.

2. Inspect allocated requests on each node to find the nodes closest to saturation.

kubectl describe node node-1 | sed -n '/Allocated resources/,/Events/p'

Expected output: CPU and memory requests with percentages, for example 72 percent CPU and 88 percent memory.

3. Simulate N-1 capacity by subtracting the largest node and checking whether the remaining requests fit.

kubectl get pods -A -o json \
  | jq '[.items[].spec.containers[].resources.requests.memory // "0"] | length'
kubectl get pods -A -o json \
  | jq -r '.items[].spec.containers[].resources.requests.memory // empty' \
  | sed 's/Mi$//' | awk '{sum += $1} END {printf "total requests: %d Mi\n", sum}'

Expected output: Total memory requests across all pods, which can be compared with allocatable capacity minus the largest node.

4. Record the headroom decision and adjust autoscaler maximum or replica counts before the next peak.

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml | grep -A 5 'Max nodes'

Expected output: The autoscaler maximum is documented, and the review states the target headroom percentage and owner.

Troubleshooting

Problem

Pods are Pending even though the node shows free memory

Solution

Requests may be fragmented across nodes or blocked by taints and topology constraints; check the scheduler event for the exact reason.

Problem

Requests are far below actual usage

Solution

Profile real usage and raise requests gradually; under-requesting hides capacity risk and lets nodes become overcommitted.

Limitations

Requests are a scheduling reservation, not actual usage; a cluster can be safe on requests and still struggle on real load.
N-1 math ignores zone spread and pod topology constraints, which can reduce usable capacity further.

Headroom is the capacity you cannot see

Allocated requests can look comfortable until a node disappears. Headroom is the difference between what the cluster can schedule and what it needs at peak, including the loss of one node or one availability zone. Without a periodic review, headroom quietly shrinks as new services add requests and the autoscaler maximum stays fixed.

Measure the right numbers

Use allocatable, not capacity, because kubelet reserves CPU and memory for system components. Sum requests from pod specs, not usage metrics, because the scheduler places pods based on requests. Then compute N-1: subtract the largest node from total allocatable and check whether the remaining requests still fit. If they do not, a single failure causes Pending pods.

kubectl get nodes -o custom-columns='NODE:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory'
kubectl describe node node-1 | sed -n '/Allocated resources/,/Events/p'

Set a target

A common target is 30 percent free memory after N-1, which absorbs burst scaling and replacement pods. CPU can run hotter because it is compressible, but memory pressure causes eviction. Record the target in the capacity review and alert when a cluster stays below it for a week.

Close the gaps

If headroom is low, options are adding nodes, raising the autoscaler maximum, reducing requests after profiling, or moving batch workloads to a separate pool. Check that the autoscaler can actually reach the maximum: account quotas, IP exhaustion, and node group limits all block scaling. Review after every major launch and before seasonal peaks. The expected outcome is a written headroom percentage for the N-1 case, a decision on each gap, and confirmation that the scheduler could place replacement pods during a simulated drain.