Kubernetes Evicted Pod — Node Pressure and Ephemeral Storage

Diagnose Kubernetes pod evictions by reading eviction messages and node pressure conditions, then fix resource requests, ephemeral storage limits, and kubelet GC.

intermediatedebuggingkuberneteseviction

Symptoms

Pods show STATUS Evicted and are not restarted
kubectl describe pod reports The node was low on resource: ephemeral-storage
Several pods on one node fail at the same time
New pods stay Pending after eviction because no capacity is freed

Possible causes

The container writes logs or temporary files beyond its ephemeral-storage limit
Node memory or disk is exhausted and kubelet evicts the lowest-priority pods
emptyDir volumes have no sizeLimit and fill the node filesystem
Image or container garbage collection thresholds leave too little free space

Diagnosis steps

List failed pods and their eviction reasons

kubectl get pods -A --field-selector status.phase=Failed -o wide

Interpretation

Evicted pods remain in Failed phase and are not recreated by their controller in place. The node column identifies whether the problem is node-specific.

Read the eviction message

kubectl describe pod -n app orders-7c9f8b6d5-2xk4p | sed -n '/Status:/,/Events:/p'

Interpretation

The message names the resource and often the offending container, for example 'The node was low on resource: ephemeral-storage. Container orders was using 4212Ki, which exceeds its request of 0.'

Check node pressure conditions

kubectl describe node ip-10-0-3-17 | grep -A8 -E 'Conditions:|Allocated resources:'

Interpretation

MemoryPressure, DiskPressure, or PIDPressure set to True explains why kubelet started evicting. The allocated resources table shows whether requests already exceed capacity.

Inspect disk usage on the node

kubectl debug node/ip-10-0-3-17 -it --image=busybox -- df -h /var/lib/kubelet

Interpretation

A filesystem above the kubelet eviction threshold confirms node-level pressure. If disk is not the issue, check container and image garbage collection settings on the node instead.

Root cause

The kubelet evicts pods to protect node stability when memory, disk, or inode usage crosses an eviction threshold. Ephemeral storage covers container writable layers, logs, and emptyDir volumes; a container that writes without a limit can consume the node filesystem even if its memory usage is fine. Evicted pods are replaced by their controller only if capacity is actually released.

Fix

Set CPU and memory requests plus ephemeral-storage requests and limits for every workload, give emptyDir a sizeLimit, and ship logs instead of writing them to the container filesystem. Raise kubelet eviction thresholds only after reducing real usage, and clean unused images or enlarge the node disk. Add priorityClasses so critical pods survive pressure.

Risk note

Evicting pods does not delete their containers or volumes automatically, so failed pods can accumulate. Setting an ephemeral-storage limit too low causes repeated eviction of a healthy app. Raising eviction thresholds risks the node running out of disk entirely and becoming NotReady.

Verify the fix

Delete the Evicted pods, confirm replacements schedule and stay Running, and verify node conditions report MemoryPressure and DiskPressure False. Monitor ephemeral-storage usage per pod for a full day and confirm no new eviction events appear.

Evicted is a node decision

An Evicted pod was terminated by the kubelet, not by its Deployment or by the scheduler. The container may have been healthy; the node needed to reclaim memory, disk, or inodes to stay available.

kubectl get pods -A --field-selector status.phase=Failed
kubectl describe pod -n app orders-7c9f8b6d5-2xk4p | grep -A4 'Status:'

The message usually names the resource and sometimes the container. ephemeral-storage is the most common surprise because many teams set CPU and memory limits but never account for logs and scratch files.

Confirm node pressure

kubectl describe node ip-10-0-3-17 | grep -A8 -E 'Conditions:|Allocated resources:'

DiskPressure=True or MemoryPressure=True is the direct cause. Look at the allocated resources table as well: if requests already exceed allocatable, the scheduler should not place more pods there, but daemonsets and unrequested pods still consume real capacity.

Fix limits and scratch space

resources:
  requests:
    cpu: 250m
    memory: 256Mi
    ephemeral-storage: 1Gi
  limits:
    memory: 512Mi
    ephemeral-storage: 2Gi
volumes:
  - name: cache
    emptyDir:
      sizeLimit: 500Mi

Ship logs to stdout and a collector instead of writing large files into the container. Add a priorityClassName for critical workloads so the kubelet evicts lower-priority pods first. If the node disk is simply too small, resize it or adjust kubelet GC thresholds deliberately.

Verify

Remove the failed pods so they stop cluttering the namespace, then confirm replacements are Running on a node with MemoryPressure=False and DiskPressure=False. Watch kubectl get events --field-selector reason=Evicted for a day; no new events means the limits and cleanup are actually working, not just that the burst passed.