Linux Too Many Open Files — Find the Leaking File Descriptor

Resolve EMFILE and Too many open files errors by counting descriptors per process, identifying the leaked resource, and sizing systemd and kernel limits correctly.

intermediatedebugginglinuxfile-descriptorssystemd

Symptoms

Too many open files, or accept failed: EMFILE
The error appears after hours of traffic and survives a restart only briefly
New connections are refused while the process is still running
ls /proc/<pid>/fd stops growing only when the service restarts

Possible causes

Sockets, files, or pipes are opened per request and never closed on error paths
The service limit LimitNOFILE is far below the expected connection count
A library opens a file per operation, such as a logging or database driver handle
inotify instances or event loop descriptors accumulate without cleanup
The system-wide file table is exhausted, producing ENFILE instead of EMFILE

Diagnosis steps

Count descriptors against the process limit

ls /proc/$(pgrep -f 'node server.js' | head -1)/fd | wc -l; grep 'Max open files' /proc/$(pgrep -f 'node server.js' | head -1)/limits

Interpretation

A count close to the soft limit explains the failures. Compare soft and hard limits to see whether systemd or PAM applies the smaller value.

Identify which descriptor types dominate

ls -l /proc/$(pgrep -f 'node server.js' | head -1)/fd | awk '{print $11}' | sort | uniq -c | sort -nr | head -20

Interpretation

Many entries pointing to sockets indicate connection leaks. Repeated regular files point to handles opened per request.

Watch the count across request cycles

pid=$(pgrep -f 'node server.js' | head -1); for i in $(seq 1 5); do ls /proc/$pid/fd | wc -l; sleep 10; done

Interpretation

A rising count that never returns to baseline is a leak. A count that oscillates and stays bounded is capacity pressure.

Check the system-wide file table

cat /proc/sys/fs/file-nr; sysctl fs.file-max

Interpretation

The first number is allocated handles and the third is the system maximum. Allocation near the maximum produces ENFILE for every process.

Root cause

A process ran out of file descriptor slots, either because its soft limit was too low for the real concurrency or because descriptors were leaked and never released. Each socket, pipe, and open file consumes one slot, and once the table is full the kernel rejects new opens with EMFILE or ENFILE before application code can react.

Fix

Fix the leak by closing descriptors on every path with finally, defer, or context managers, then raise LimitNOFILE in the systemd unit and verify the hard limit. Use connection pooling with a hard cap so concurrency cannot exceed the descriptor budget, and raise fs.file-max only when the whole system is saturated. Restarting the service only buys time.

Risk note

Raising limits without fixing a leak delays the failure and can exhaust kernel memory. Changing /etc/security/limits.conf does not affect an already running systemd service, and ulimit in a shell does not change the unit limit. Never set fs.file-max arbitrarily high on a small host.

Verify the fix

Replay the request mix that triggered the error and sample the descriptor count after each cycle. The count should return to its baseline instead of climbing, the soft limit should exceed peak usage with headroom, and no EMFILE or ENFILE entries should appear in the logs.

One slot per descriptor

Every socket, pipe, inotify watch, and open file consumes a file descriptor. Linux enforces two limits: a per-process soft and hard limit, and a system-wide table. Exceeding the per-process limit returns EMFILE; exhausting the system table returns ENFILE. The message is usually Too many open files in both cases, so the first question is scope.

Count, then classify

pid=$(pgrep -f 'node server.js' | head -1)
ls /proc/$pid/fd | wc -l
grep 'Max open files' /proc/$pid/limits
ls -l /proc/$pid/fd | awk '{print $11}' | sort | uniq -c | sort -nr | head

A count hugging the soft limit is the immediate cause. The link targets in /proc/<pid>/fd show whether the descriptors are sockets, regular files, or anon inodes. Sampling the count every few seconds across request cycles separates a leak from simple concurrency: a leak never returns to baseline, while capacity pressure rises and falls.

cat /proc/sys/fs/file-nr prints allocated handles, unused-but-allocated handles, and the maximum. For a single noisy service, the per-process limit is the relevant ceiling; system-wide exhaustion shows the same error for unrelated processes.

Fix the leak and the budget

Close resources on every path, including exceptions and early returns. A connection pool with a hard maximum keeps concurrency inside the descriptor budget instead of relying on the limit as a safety net.

# /etc/systemd/system/app.service.d/limits.conf
[Service]
LimitNOFILE=65535

sudo systemctl daemon-reload
sudo systemctl restart app.service
systemctl show app.service -p LimitNOFILE

Raise the limit only after checking peak usage and headroom. A restart masks a leak for exactly as long as the leak rate predicts.

Verify

Run the workload that previously failed and sample ls /proc/$pid/fd | wc -l after each cycle. The count must settle back to baseline, peak usage must stay below the soft limit with margin, and the logs must stay free of EMFILE and ENFILE entries.