Python — Memory Grows Despite Freed Objects: gc Cycles

Investigate Python memory growth that survives manual del, find reference cycles with gc.get_referrers and objgraph, and break cycles with weakref or explicit close.

advanceddebuggingpythonmemorygc

Symptoms

RSS climbs steadily across request cycles even though tracemalloc current resets to baseline after each request
gc.get_count shows gen2 collections happening but memory does not return
Objects with __del__ involved in cycles are never collected (gc.garbage list grows)

Possible causes

Bidirectional references between long-lived registries and per-request objects create cycles the generational GC handles late or never
Instances define __del__, which before Python 3.4 (and with certain patterns today) prevents cycle collection into gc.garbage
Caches hold keys and values that reference each other plus the cache itself
Exception objects retain local frames through their traceback, forming implicit cycles with handler closures

Diagnosis steps

Count objects by type across cycles

python -c "import objgraph; objgraph.show_growth(limit=10)" # run before/after a batch

Interpretation

A type whose count grows linearly with batches identifies the leaking population; flat counts mean retention lives elsewhere.

Find the cycle owners

python -c "import gc; objs=[o for o in gc.get_objects() if isinstance(o, MyType)]; print(gc.get_referrers(objs[0])[:3])"

Interpretation

The referrer list exposes what keeps the object alive — a registry, a cache, or a traceback closure.

Check the uncollectable list

python -c "import gc; gc.collect(); print(len(gc.garbage))"

Interpretation

A non-empty gc.garbage after a full collect means cycles with __del__ that the collector will not touch.

Root cause

Reference counting frees objects immediately only when no references remain. Cycles keep every member count above zero, so collection depends on the cyclic garbage collector running at the right generation, and cycles containing __del__ can be parked in gc.garbage permanently. Long-lived containers that join cycles with per-request objects convert transient data into permanent retention.

Fix

Break cycles at design level: weakref for back-pointers (parent links, caches of instances), explicit close or context managers that clear containers, avoiding __del__ in favor of context managers, and clearing exception tracebacks when storing exceptions.

Risk note

Dropping __del__ changes cleanup guarantees for resource owners. Replace it with explicit context manager protocols, not with nothing.

Verify the fix

After the fix, run repeated batches and assert object counts by type return to baseline and RSS plateaus; assert gc.garbage stays empty after full collections.

Symptom walkthrough

The paradox is the clue: you del every object, tracemalloc says current is back to baseline, yet RSS climbs request over request. Instrument the population instead of the bytes:

import objgraph
objgraph.show_growth(limit=10)
# func_with_closure   1250  +1250
# RequestContext      1250  +1250

A type growing linearly with request count is the leaking cohort. Then interrogate one member for its keepers:

import gc
sample = [o for o in gc.get_objects() if isinstance(o, RequestContext)][0]
print(gc.get_referrers(sample)[:3])
# [dict of registry, <frame ...>, the handler closure]

Those referrers are the cycle edges you must cut.

Fix in practice

Cut the cycle edges by design. Back-pointers become weak references:

import weakref

class Node:
    def __init__(self, parent):
        self.parent = weakref.ref(parent)  # does not keep parent alive
        self.children = []

Registries hold instances by weak value so entries evaporate with the object:

from weakref import WeakValueDictionary

instances = WeakValueDictionary()  # id -> instance, no retention

Containers clear themselves explicitly, and resource owners move from del to the context manager protocol:

class Session:
    def close(self):
        for child in self.children:
            child.parent = None   # break cycles deterministically
        self.children.clear()

    def __enter__(self): return self
    def __exit__(self, *exc): self.close()

When storing exceptions, truncate the frame chain — exc.traceback = None — so the traceback cannot pin the handler closure into a cycle.