Linux OOM killer — Identify the Killed Process and Prevent It

Diagnose Linux OOM kills with dmesg and journalctl, identify the victim and cgroup limit, then fix leaks, concurrency, and memory sizing instead of disabling overcommit.

intermediatedebugginglinuxmemory

Symptoms

A process disappears without a trace or application-level error
dmesg shows Out of memory: Killed process with a PID and total-vm
The service restarts repeatedly under load
SSH or cron jobs are killed while the system stays partially responsive

Possible causes

A memory leak or unbounded cache grows until the kernel reclaims memory by killing a process
A container or systemd unit has a memory limit lower than the application's real working set
Too many concurrent workers multiply per-process memory beyond physical RAM
The workload relies on overcommit and the kernel cannot satisfy a large allocation

Diagnosis steps

Read the kernel OOM report

dmesg -T | grep -iE 'out of memory|killed process|oom' | tail -30

Interpretation

The report names the victim, its total-vm and rss, and often the triggering allocation. The oom_score of candidates shows why that process was chosen.

Confirm via the journal if dmesg is rotated

journalctl -k --since "1 hour ago" | grep -iE 'oom|killed process' | tail -30

Interpretation

The journal keeps kernel messages across dmesg buffer wraps and adds timestamps useful for correlating with application logs.

Check memory and cgroup pressure

cat /proc/meminfo | head -10 && systemd-cgtop -m --iterations=1 | head -15

Interpretation

MemAvailable near zero confirms pressure. A cgroup with usage at its limit explains a kill even when the host has free memory.

Find the largest consumers before the next kill

ps aux --sort=-%mem | head -15 && cat /proc/$(pgrep -n java)/oom_score

Interpretation

A process with growing RSS and a high oom_score is the likely next victim. oom_score_adj can protect critical services, but it does not create memory.

Root cause

The Linux kernel cannot satisfy a memory allocation and invokes the OOM killer to free pages. It selects a victim by oom_score, which weighs memory usage and adjustments, so the largest or least protected process is usually killed. Container and systemd memory limits trigger the same mechanism inside a cgroup, even when the host has available RAM, which is why a service can be killed while the node looks healthy.

Fix

Fix the leak or bound the cache, reduce worker concurrency, and size container limits from measured peak RSS with headroom. Protect critical services with oom_score_adj or systemd MemoryMin, and make batch jobs nice or cgroup-limited so they cannot starve interactive workloads. Add swap or increase memory only when the working set is genuinely larger than RAM, and use earlyoom or a kernel OOM handler to log before the kill.

Risk note

Setting oom_score_adj to -1000 can make a leaking process unkillable and hang the system. Disabling overcommit risks allocation failures elsewhere and is rarely the right fix. Adding swap can turn a fast kill into prolonged thrashing, especially for latency-sensitive services.

Verify the fix

Run the same workload and confirm MemAvailable stays above the safety margin and no new OOM entries appear in dmesg or the journal. Check the victim process RSS peak stays below its cgroup limit, and verify the service remains responsive through the load test.

The kernel killed it, not the app

Out of memory: Killed process 4821 (java) total-vm:8123456kB, anon-rss:4021760kB
oom_reaper: reaped process 4821 (java), now anon-rss:0kB

The process did not crash and usually left no application log. The kernel needed pages and chose a victim by oom_score.

Confirm the victim and the pressure

dmesg -T | grep -iE 'out of memory|killed process' | tail -30
journalctl -k --since "1 hour ago" | grep -iE 'oom|killed process'
cat /proc/meminfo | head -10
systemd-cgtop -m --iterations=1 | head -15

MemAvailable near zero means host pressure. A cgroup at its memory.max explains a kill while the host still has free memory, which is common in containers and systemd services.

Reduce the demand

Fix the leak or cap the cache; RSS that grows with uptime is not a sizing problem.
Lower worker or thread counts so peak concurrency fits the limit.
Set container memory limits from measured peak RSS plus headroom.
Protect critical units with oom_score_adj or MemoryMin; do not set -1000.
Limit batch jobs so they cannot consume all pages.

systemctl show -p MemoryMax -p MemoryCurrent app.service
cat /proc/$(pgrep -f app.service)/oom_score

Verify

Re-run the load and watch MemAvailable, cgroup usage, and the journal. No new OOM entries should appear, and the process RSS peak should stay below its limit. If the kill returns, capture the allocation pattern with smem or a heap profile before adding memory; otherwise the next spike will simply need more.