Memory Leak Hunt Workflow

Confirm growth is a leak with trend analysis, bisect the retained set with heap snapshots, and verify the fix by flatness under soak load.

advanced~40 minperformance-tuningmemorydebugging

Objective

Find and fix the specific retention causing memory growth, verified by a flat heap under sustained load rather than by hope.

Environment

Heap snapshot tooling (Node inspector, JVM dumps, or language equivalent)
A load generator for sustained soak testing
The ability to snapshot production or production-like processes

Steps

1. Confirm it is a leak, not a cache

Plot memory over time under steady load: a leak grows monotonically, a cache plateaus, GC pressure oscillates — the shapes dictate the hunt.

Expected state: A trend plot exists classifying the growth as leak, cache, or pressure.

2. Snapshot at two points in the growth

Capture heap snapshots at the same phase of the workload separated by growth, comparing retained sizes between them.

Expected state: A delta of objects exists whose retained size grows with the trend.

3. Bisect the retaining path

Walk the retainer chain of the growing delta: which map, listener registry, or closure holds the objects alive beyond their usefulness.

Expected state: The specific retainer is named — the field, listener, or collection responsible.

4. Fix the retention mechanism

Remove the retention: unregister the listener, bound the map with LRU, close the resource, or clear the timer holding the closure.

Expected state: A new snapshot pair shows the previously growing delta now flat.

5. Verify with a soak run

Run sustained load for hours pre- and post-fix, verifying the heap curve stays flat and the process survives the intended uptime.

Expected state: The post-fix soak holds memory flat through a full intended-uptime simulation.

Expected outcome

The leak traced to its specific retainer, fixed, and verified by flat memory under soak, with the diagnostic technique retained for the next hunt.

Verification

Heap snapshots after the fix show the delta class no longer growing; the soak run completes intended uptime with flat memory.

Snapshots at the same phase, compared for the delta

Memory hunts fail from comparing apples to oranges. Snapshots taken at the same workload phase, separated only by growth, show exactly which object population expanded — and the retainer chain names who is keeping them alive.

trend (steady 200 rps, 1h):  RSS 380 -> 742 MB  monotonic = LEAK

snapshot A (after 10 min)  vs  snapshot B (after 60 min)
  delta, by retained size:
    Session objects      +38,114  (4.1MB -> 212MB)
    retainer chain:
      Session -> sessionCache Map -> disconnect handler
      -> socket close listener never removed

fix: registry.unbind(socket) in close handler
  + LRU bound on sessionCache (10k)

snapshot C vs D (post-fix, same protocol):
  Session delta: +41  (noise)      FLAT

soak (intended uptime 72h): RSS 380 -> 391MB  PASS

The retainer matters more than the object

Knowing that Session objects grow tells you what is leaking; the retainer chain tells you why. The socket close handler that never unbinds is the bug — the sessions are victims. Fixing the object (smaller sessions) would leave the mechanism broken and the leak alive.

Soak is the only honest verdict

A fix that flattens snapshots for ten minutes can still leak at a slower rate, or burst under a phase the snapshots missed. Running the intended uptime under sustained load is the only test that answers the operational question: will this process survive its deployment window?