Linux: zombie processes (defunct) accumulating

Diagnose and reap Linux zombie processes by finding the parent that fails to call wait, and fixing signal handling or daemon design.

intermediatedebugginglinuxprocesses

Symptoms

ps shows processes in defunct state or with a <defunct> suffix
The process table fills with zombies over time
A supervisor reports many unreaped children

Possible causes

The parent never calls wait or waitpid for its children
A signal handler for SIGCHLD is missing or misconfigured
A daemon double-forks without reaping intermediates
A service manager is not notified of child exits

Diagnosis steps

List zombies and find their parents

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /Z/ {print}'

Interpretation

The PPID identifies the process that should reap them; killing the zombie itself is impossible.

Inspect the parent code for wait calls

Interpretation

A parent that ignores SIGCHLD or never waits leaves terminated children in the table.

Check whether signal handling is set to SIG_IGN

Interpretation

Setting SIGCHLD to SIG_IGN can auto-reap on some systems but breaks wait-based designs.

Root cause

A zombie is a terminated child whose exit status has not been collected. The kernel keeps a minimal entry until the parent calls wait or waitpid (or the parent exits, reparenting it). Zombies consume little memory but a PID slot each, so an unreaping parent eventually exhausts the PID space.

Fix

Make the parent call wait or waitpid for every child, handle SIGCHLD to trigger reaping, or use a proper supervisor. In containers, run a minimal init such as tini so orphaned children are reaped. Where SIGCHLD is ignored deliberately, ensure the design matches the platform behaviour and still prevents accumulation.

Verify the fix

After the fix, watch ps for zombies over time; the count stays at zero, and the parent process exits its children cleanly with no defunct entries remaining.

Symptoms

ps lists processes in defunct state. They cannot be killed because they are already dead; only the parent can remove them by reaping.

Diagnosis

Find zombies and their parents:

ps -eo pid,ppid,stat,cmd | awk '$3 ~ /Z/'

The PPID points at the process that failed to wait.

Fix

Reap children in the parent:

signal(SIGCHLD, SIG_IGN);   // auto-reap, platform-dependent behaviour
// or explicitly
while (waitpid(-1, NULL, WNOHANG) > 0) { }

In containers, add a minimal init:

ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]

Verification

Monitor the process table; no defunct entries appear as children terminate, confirming the parent now reaps them and PID slots are reclaimed.

Zombies as a Signal About Design

A handful of transient zombies is normal and harmless, but a growing count is a design smell: the service spawns children it never collects. That often indicates code that shells out frequently, a supervisor that ignores SIGCHLD, or a container running as PID 1 without an init. The fix belongs in the parent, since a zombie cannot be killed from outside. Adding a minimal init in containers, reaping in the parent, and avoiding unnecessary child processes all prevent the eventual PID exhaustion that turns a cosmetic warning into an outage.