Python — OSError: [Errno 24] Too many open files

Trace Python file descriptor leaks with lsof and fd counts, find open calls missing with or close, and switch to context managers everywhere.

beginnerdebuggingpythonfile-ioresource-leak

Symptoms

OSError: [Errno 24] Too many open files raises after the process runs for a while, always at an open or socket call
lsof -p shows thousands of open files, mostly regular files or pipes pointing at the same paths
The failure count correlates with the number of requests or files processed, not with time alone

Possible causes

open() calls in loops or request handlers without with blocks or close()
tempfile objects whose file handle is kept while only the name is used afterward
subprocess pipes whose stdout and stderr are never read to completion or closed
An early return or exception path skips the close call that sits at the end of the function

Diagnosis steps

Count descriptors and group them

ls /proc/<pid>/fd | wc -l && lsof -p <pid> | awk '{print $NF}' | sort | uniq -c | sort -rn | head

Interpretation

A fd count in the thousands, dominated by repeats of the same few paths, identifies exactly which resource leaks.

Correlate growth with work

watch -n5 "ls /proc/<pid>/fd | wc -l" # while running traffic

Interpretation

fd count tracking request count means per-request leakage; growth only over wall time points at timers or logs.

Audit the leak paths in code

grep -Rn "open(" src/ | grep -v "with open" | head -20

Interpretation

Every open without a with is a candidate, and ones inside try blocks with early returns are the usual offenders.

Root cause

File descriptors are a finite per-process resource. Each open, pipe, or socket that is never closed consumes one permanently. Under steady traffic the count climbs until the kernel refuses the next open with EMFILE, which Python reports as OSError Errno 24.

Fix

Wrap every file, pipe, and temporary file in a with block so close is structural rather than manual; use contextlib.closing for objects exposing close without context manager support; read and close subprocess pipes deterministically; and run tests with a low RLIMIT_NOFILE to make leaks fail fast.

Risk note

Raising ulimit -n only widens the bucket. The process still leaks and will hit the new ceiling later, with a bigger blast radius.

Verify the fix

Run a fixed number of requests and assert the fd count returns to its pre-run baseline exactly; repeat with a deliberately lowered ulimit to prove the guard fails loudly.

Symptom walkthrough

The errno is precise and the growth pattern confirms it:

OSError: [Errno 24] Too many open files: 'data/2026-10-22.ndjson'

Watch the descriptor directory live while traffic flows:

watch -n5 "ls /proc/$(pgrep -f app)/fd | wc -l"
# 1042 -> 1089 -> 1137 ...  <- one per request

Group the handles by path to name the leaking resource directly:

lsof -p $(pgrep -f app) | awk '{print $NF}' | sort | uniq -c | sort -rn
#  489 data/2026-10-22.ndjson

Hundreds of handles to the same file is the receipt: the code opens it per request and closes it on no path.

Fix in practice

Make closing structural with the with statement, which survives exceptions and early returns:

with open(path, encoding="utf-8") as f:
    for line in f:
        process(line)
# closed here on every path, including raise

For subprocess pipes, consume and close both streams deterministically:

with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as p:
    out, err = p.communicate()   # reads to EOF, closes both fds

For objects that only expose close, delegate to contextlib.closing. In tests, lower the ceiling so leaks fail within seconds instead of hours:

import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (256, 256))