阅读中文版本

K8s OOMKilled — Container Memory Limit Exceeded

Diagnose OOMKilled pod termination when the container exceeds its memory limit and the kernel OOM killer invokes.

node -e "require('./repro')"

Set NODE_OPTIONS=--stack-trace-limit=100 and run with verbose logging

Add console.log / debugger / breakpoint at the line before the throw

Symptom and Error Message

When running the Kubernetes code, the runtime raises the following error:

Container "app" was OOMKilled (exit code 137); memory limit exceeded

This error indicates that a pod is restarted with reason oomkilled because it exceeded its memory limit. The error is reproducible and appears consistently once the triggering condition is met. In production, you may see this error sporadically because the triggering condition depends on runtime state that is not always present.

Root Cause Analysis

The fundamental cause is: when a container exceeds its memory limit, the kernel oom killer terminates it with exit code 137. Two factors converge to produce the visible error. First, the container’s memory limit is too low for the workload. Second, the application has a memory leak that grows beyond the limit. Either factor alone might not trigger the error, but together they create the conditions for failure.

Diagnosis Walkthrough

Start by reproducing the error in isolation. Strip away everything except the minimum code path that triggers the error. Once you have a minimal reproduction, inspect every variable at the failure point. Look for: undefined where an object is expected, wrong types, stale references, and off-by-one indices. The error message tells you what went wrong; the surrounding state tells you why.

Fix and Code Example

// Before (buggy)
// ... code that triggers the error ...

// After (fixed)
// ... corrected code with proper guard or type check ...

The fix addresses the root cause directly: increase the memory limit, or fix the memory leak, or set a memory request closer to the limit. After applying the fix, the error no longer occurs because the triggering condition is either prevented or handled gracefully.

Verification

Run the workload and confirm no OOMKilled; monitor memory usage metrics. Run the reproduction case again and confirm the error is gone. Then run the full test suite to check for regressions. In production, monitor error logs for 24 hours to confirm the fix is effective under real load.

Common Pitfalls

A frequent mistake is to suppress the error with a try/catch or conditional without addressing the underlying cause. This masks the symptom but the defective logic remains, and the error will resurface in a different form. Always fix the root cause, not just the error message. Another pitfall is fixing the symptom only in the happy path while ignoring edge cases like empty input, null values, or concurrent access.