Kubernetes OOMKilled — Container Exceeded Memory Limit

Fix OOMKilled by measuring real memory use, raising or right-sizing limits, and separating a leak from a transient spike.

intermediatedebuggingkubernetesmemoryresources

Symptoms

Pod restarts repeatedly with last state "OOMKilled" and exit code 137
The container dies under load but runs fine when idle
kubectl describe pod shows "Reason: OOMKilled"

Possible causes

The container memory limit is below its real working set
An unbounded cache or buffer grows until the cgroup limit is hit
A memory leak accumulates across requests

Diagnosis steps

Confirm OOMKilled and the limit

kubectl describe pod <pod> | grep -A 5 "Last State\|Limits"

Interpretation

Last State reason OOMKilled plus the memory limit shows the ceiling that was hit.

Measure actual usage against the limit

kubectl top pod <pod> && cat /sys/fs/cgroup/memory.max

Interpretation

Comparing live usage to the cgroup limit tells you whether to raise the limit or fix a leak.

Root cause

The kernel OOM killer terminates a container whose cgroup memory usage reaches its limit, killing the largest process. Kubernetes reports exit code 137 and reason OOMKilled, which is distinct from application-level errors.

Fix

Right-size the memory limit to observed peak plus headroom, fix leaks or unbounded caches, set requests to inform the scheduler, and use a memory-limit-aware runtime (for example a heap cap) so the app manages its own memory.

Risk note

Setting limits far too high risks node-level pressure and eviction; base values on measured peaks, not guesses.

Verify the fix

Run the original load and confirm no OOMKilled restarts; verify steady-state memory stays below the limit with headroom.

Symptoms

This debugging guide addresses “Kubernetes OOMKilled — Container Exceeded Memory Limit”. Start from the symptoms, follow the ordered diagnostics, and confirm each hypothesis before moving on; the fix and verification steps then close the loop.

Observed symptoms include:

Pod restarts repeatedly with last state “OOMKilled” and exit code 137
The container dies under load but runs fine when idle
kubectl describe pod shows “Reason: OOMKilled”

Reproduce and Diagnose

Reproduce the failure in isolation before changing any code. Work through the checks below in order; each one narrows the search space.

Step 1: Confirm OOMKilled and the limit

kubectl describe pod <pod> | grep -A 5 "Last State\|Limits"

Last State reason OOMKilled plus the memory limit shows the ceiling that was hit.

Step 2: Measure actual usage against the limit

kubectl top pod <pod> && cat /sys/fs/cgroup/memory.max

Comparing live usage to the cgroup limit tells you whether to raise the limit or fix a leak.

Likely contributors, in order of frequency:

The container memory limit is below its real working set
An unbounded cache or buffer grows until the cgroup limit is hit
A memory leak accumulates across requests

Root Cause

The Fix

Watch out: Setting limits far too high risks node-level pressure and eviction; base values on measured peaks, not guesses.

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"   # peak + headroom, based on measurement
    cpu: "500m"

# Node.js: cap the heap below the container limit
env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"

Verify and Prevent

Run the original load and confirm no OOMKilled restarts; verify steady-state memory stays below the limit with headroom.

Once verified, add a regression test that reproduces the original symptom so the failure cannot silently return.

This class of failure is usually environmental rather than pure logic, so capture the exact command, the environment variables, and the working directory in your bug report. Reproduce it in the smallest possible script, then widen the scope only once the minimal case is understood. A short written checklist for this failure mode saves hours the next time it appears.