Tune Node Heap and GC Settings from Evidence

1. Record GC events and heap stats under a fixed load window.

node --trace-gc --trace-gc-verbose dist/server.js 2>gc.log & LOAD=200rps; sleep 300; kill %1
grep -c "Scavenge" gc.log; grep -c "Mark-Compact" gc.log; grep "Mark-sweep\|Mark-Compact" gc.log | tail -5

Expected output: Event counts print, for example 4,100 scavenges versus 22 major GCs, and the last major collections show their pause durations.

2. Chart heap growth between major collections.

grep -oE "[0-9.]+ ms.*] [0-9]+ -> [0-9]+ \([0-9]+\) Mb" gc.log | awk '{print $1, $NF}' | head -20 || grep -oE "heap [0-9]+ -> [0-9]+ " gc.log | head -20

Expected output: Heap-before values print per major GC; a sawtooth rising from 300 to 900 MB before each collection flags premature promotion of short-lived objects.

3. Quantify GC pause impact on request latency.

jq -r 'select(.route == "/api/search") | [.at, .durationMs] | @tsv' requests.log | awk '{if ($2 > 200) print $1, "latency", $2}' | head -8

Expected output: Slow-request timestamps line up with major GC timestamps from gc.log, correlating pauses with user-visible stalls.

4. Apply the minimal flag set matching the diagnosis.

# diagnosis: many scavenges + fast old-gen growth -> objects survive
# two scavenges by being allocated before a request completes

# fix A: bigger young generation, fewer premature promotions
node --max-semi-space-size=64 dist/server.js

# fix B: cap old space so majors happen often but cheaply
node --max-old-space-size=1024 dist/server.js

# fix C (diagnosis was fragmentation): compact on every major
node --expose-gc --max-old-space-size=1024 \
  -e "setInterval(() => { if (global.gc) global.gc(); }, 30000).unref?.(); import('./dist/server.js')" 2>/dev/null || node --max-old-space-size=1024 --gc-interval=1024 dist/server.js

Expected output: Under the same 200 rps load, major GC pauses drop from hundreds of milliseconds to tens, and the RSS ceiling stops creeping.

5. Re-run the identical measurement to verify and keep the numbers.

node --max-semi-space-size=64 --trace-gc dist/server.js 2>gc2.log & sleep 300; kill %1; echo "scavenges: $(grep -c Scavenge gc2.log)  majors: $(grep -cE 'Mark-(Compact|sweep)' gc2.log)"; grep -oE "\] [0-9]+ -> [0-9]+ \([0-9]+\) Mb" gc2.log | tail -3

Expected output: The after-numbers print alongside the before-numbers, documenting the flag decision with the evidence that justified it.

Problem

RSS keeps growing despite flat heap

Solution

Memory lives outside V8 heap: Buffers, workers, or native addons; track external memory via process.memoryUsage().external before touching heap flags.

Problem

Bigger semi-space makes latency worse

Solution

The workload allocates long-lived objects per request; larger young generation delays promotion and grows the scavenge cost; revert and fix allocation sites or cap old space instead.

Problem

Major GC pauses persist after tuning

Solution

Large long-lived graphs dominate; reduce retained data (caches with TTL, LRU bounds), because flags cannot shrink the live set your code insists on holding.