Python — High RSS After Processing Large Datasets

Explain why Python RSS stays high after big objects are freed, separate allocator arena retention from true leaks with tracemalloc, and release pages deliberately.

advanceddebuggingpythonmemoryperformance

Symptoms

RSS stays several gigabytes above baseline long after the job that processed the large dataset finished
tracemalloc shows current memory back near baseline while ps or container metrics do not
The process passes leak tests (object counts, gc.get_objects) yet container OOM alerts still fire

Possible causes

CPython arena fragmentation: freed arenas with a single live object keep whole 256KB pages mapped
glibc malloc retains freed chunks in the heap top rather than returning them with munmap
Native extensions (numpy BLAS buffers, tokenizer pools) allocate outside the Python heap and free on their own schedule
The OS page cache and lazy reclamation inflate container memory accounting

Diagnosis steps

Compare Python view and OS view

python -c "import tracemalloc,gc; gc.collect(); print(tracemalloc.get_traced_memory())" && ps -o rss= -p $$

Interpretation

A traced current near baseline with high RSS means Python objects are freed and the gap belongs to the allocator or native layers.

Check fragmentation at the arena level

python -c "import sys; print(sys.version)" # and run with PYTHONMALLOC=debug to annotate arenas

Interpretation

Arena-level retention shows as RSS that does not drop after gc.collect(); object counts dropping proves the objects themselves are gone.

Test deliberate release behavior

python -c "import numpy as np; a=np.ones((300_000_000//8,)); del a; import gc; gc.collect(); print('freed')" && ps -o rss= -p $!

Interpretation

Allocating and freeing a large array then observing RSS tells you whether the platform reclaims immediately or retains the arena.

Root cause

CPython and the underlying C allocator do not necessarily return freed memory to the OS. Freed arenas stay mapped when fragmented, glibc holds freed chunks in its heap, and native extension buffers free asynchronously. The process holds no live Python objects, yet RSS stays high because pages remain committed on the process behalf.

Fix

Batch work in child processes that exit and release everything, call malloc_trim on glibc between large phases, reuse large buffers instead of repeatedly allocating, and size container limits to the working set plus measured retention rather than to the Python object total.

Risk note

Forcing malloc_trim in a loop adds syscalls and can fight the allocator. Call it at phase boundaries only.

Verify the fix

After the fix, a full workload cycle ends with RSS within a bounded band above baseline across repeated runs, and container OOM alerts stay silent.

Symptom walkthrough

The contradiction between two meters is the whole clue:

tracemalloc current:  182 MiB   <- Python objects: freed
gc.get_objects:       back to baseline
container RSS:        4.1 GiB   <- pages still committed

When the Python view says the objects are gone but the OS view disagrees, the memory lives below the interpreter: in allocator arenas, in glibc heap retention, or in native extension buffers. Separate the layers before calling anything a leak — a true leak keeps object counts high; retention keeps only pages.

Fix in practice

The structural fix is process isolation for large phases:

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor(max_workers=1) as pool:
    result = pool.submit(load_and_transform, path).result()
# worker exits: every page returns to the OS, guaranteed

Within a long-lived process, trim at phase boundaries on glibc:

import ctypes

libc = ctypes.CDLL("libc.so.6")
libc.malloc_trim(0)  # return free heap chunks to the OS

Reuse large buffers instead of churning them:

buf = bytearray(CHUNK)          # allocated once
for chunk in read_chunks(path):
    buf[:len(chunk)] = chunk     # reused, no new arena each time

And set container memory limits from measurement — steady-state RSS plus retention band — not from Python-object accounting, or the OOM killer will keep surprising you on days the data is larger.

Verification

　　Run the full workload three times consecutively in the same process and record RSS after each cycle. Acceptance: the first cycle peaks and partially retains, subsequent cycles reuse the retained pages instead of growing further, and final RSS stays inside a fixed band above baseline. Then run the process-pool variant and assert RSS after each batch returns to baseline exactly. Keep both numbers in the deploy notes so capacity planning uses measured reality rather than the interpreter optimistic view.