Detect Memory Leaks with a Soak Test

Drive repeated workload cycles for hours, plot heap against cycle count, and distinguish a leak from a plateau before blaming the runtime.

~40 mintestingperformanceobservability

Task

Establish whether a service leaks memory under sustained load, quantify the leak rate, and verify the fix with the same harness.

When to use this

The container restarts every few days with OOM. Dashboards show memory climbing, but nobody can say whether it is a leak or a slow cache filling.

Prerequisites

A load generator able to run for hours
Heap metrics or process RSS exposed from the service
A graphing tool or the ability to eyeball CSV

Steps

1. Run the workload in long cycles while recording memory.

for i in $(seq 1 2000); do curl -s -o /dev/null "http://localhost:3000/orders?page=$((i % 40))"; done & watch -n 60 "curl -s localhost:9090/metrics | grep process_resident_memory_bytes" | tee soak.log

Expected output: RSS samples every minute accumulate into a timeline long enough to show the trend shape.

2. Plot memory against cycles, not wall-clock time.

x = cycle_count     # every 100 requests
y = rss_mb
# leak:    y = a + b*x with b > 0
# plateau: y approaches an asymptote

Expected output: A straight rising line against cycles is a leak; a curve flattening to an asymptote is a cache warming.

3. Force GC between cycles to separate garbage from retention.

if (global.gc) global.gc();   // run with --expose-gc
const rss = process.memoryUsage();
log({ rss: rss.rss, heapUsed: rss.heapUsed, cycle });

Expected output: Post-GC heap still rising across cycles means retained objects: a leak, not collection lag.

4. Diff two heap snapshots at steady state.

node --inspect server.js & curl -s -X POST localhost:3000/admin/heap-snapshot -d '{"label":"after-1000"}'; ls -la *.heapsnapshot

Expected output: Two snapshots separated by known cycles, diffed in the profiler, name the retained allocation sites.

5. Fix the dominant retainer and re-soak.

grep -n "new Map\|addListener\|setInterval" src/cache.js | head -5

Expected output: The usual suspects, an unbounded cache, a listener added per request, an interval never cleared, appear at the top of the diff.

Troubleshooting

Problem

Memory rises then plateaus, is it a leak

Solution

A plateau is expected behavior: caches, connection pools, and JIT warmup all settle; only a slope against cycles after steady state indicates a leak.

Problem

Leak only visible after days

Solution

Shrink the cycle and raise the rate; leaks scale with operations, so 10x traffic exposes a days-long leak in hours.

Problem

Snapshot tooling unavailable in the runtime

Solution

Fall back to RSS slope plus code inspection of allocation sites; container OOM events provide the confirming endpoint.

Limitations

Soak tests measure the tested workload; unexercised code paths can still leak.
RSS includes allocator overhead and fragmentation, which can grow without a true object leak.

Slope against cycles is the signature

Memory climbing in a dashboard is ambiguous: caches fill, pools grow, runtimes warm up. The disambiguator is the x-axis. Plot memory against completed workload cycles, force GC between samples, and a leak becomes unmistakable: post-GC heap rising linearly with cycles. The same plot after the fix flattens, and the soak harness becomes the regression gate.

# Sample RSS every minute while cycling the workload
for i in $(seq 1 2000); do
  curl -s -o /dev/null "http://localhost:3000/orders?page=$((i % 40))"
done &
watch -n 60 "curl -s localhost:9090/metrics | grep process_resident_memory_bytes" | tee soak.log

Force the GC before judging

A rising line without forced collection proves only that allocation rate exceeds collection timing, which is normal. Sampling after an explicit GC isolates what is retained: if post-GC heap still climbs with cycles, objects are being kept alive by references the application owns. That distinction converts an ominous dashboard into an actionable finding.

Cycles compress days into hours

A leak that OOMs in four days at production traffic will OOM in an hour at ten times the rate, because leaks are proportional to operations, not time. Scale the soak workload until the slope is measurable within a CI-friendly window, and the same acceleration runs as a regression gate on every release.