Node.js — Heap Out of Memory Caused by Missing Stream Backpressure

Diagnose heap out of memory while streaming large responses or files, spot the writable that never waits for drain, and add backpressure with pipeline and highWaterMark.

advanceddebuggingnodejsstreamsmemory

Symptoms

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory during a file or export job
process.memoryUsage().heapUsed climbs in lockstep with the size of the input, even though the code streams chunk by chunk
The service survives small exports and dies only on multi-gigabyte inputs

Possible causes

A readable is piped or looped into a writable that always accepts chunks, so writes queue in memory faster than they flush
A manual while loop reads chunks without awaiting the writable drain event
res.write or fs.write return values are ignored, so false is never acted on
An intermediate transform accumulates state (for example concatenating a string) instead of passing chunks through
A promise-based wrapper collects all chunks into an array before responding

Diagnosis steps

Confirm the heap growth pattern

node --expose-gc --inspect script.js # then take heap snapshots in DevTools

Interpretation

Two snapshots dominated by Array or String retention inside the streaming path confirm buffering rather than a leak from listeners or closures.

Audit every write call for drain handling

grep -R "write(\|pipeline(\|pipe(" src/ | grep -v drain

Interpretation

Any hot path that writes without checking the return value or using pipeline is a backpressure suspect.

Watch the writable length live

setInterval(() => log(out.writableLength), 500)

Interpretation

writableLength growing without bound means the stream is queuing chunks in memory instead of applying pressure upstream.

Root cause

The readable side produced data faster than the writable side could flush, and the code never propagated the false return value or drain event. Every unread chunk was buffered inside the writable stream queue, so effective memory usage tracked the full input size despite the streaming design.

Fix

Replace manual read/write loops with stream.pipeline or Readable.pipe with proper error handling, await the drain event when writing by hand, size highWaterMark to the chunk budget you can afford, and make transforms stateless so they never accumulate the whole payload.

Risk note

Raising --max-old-space-size only delays the crash and hides the symptom; a 4 GB heap on a 6 GB input still dies. Fix the buffering first.

Verify the fix

Stream a file several times larger than the heap limit and assert that heapUsed stays under a fixed ceiling and writableLength returns to near zero between flushes.

Symptom walkthrough

The crash message is generic, but the trigger is specific: it happens while moving a large payload through code that looks streamed:

FATAL ERROR: Reached heap limit Allocation failed -
JavaScript heap out of memory

Take two heap snapshots a few seconds apart. When the retained objects are mostly Array buffers owned by a writable stream, the readable side is outrunning the sink. A quick live check makes it undeniable:

setInterval(() => {
  console.log('queued bytes', out.writableLength);
}, 500);

A bounded value that returns to zero means healthy backpressure. A value that grows toward the input size means every chunk is parked in memory.

Fix in practice

Let pipeline own the plumbing, because it wires destroy, error propagation, and drain handling for you:

import { pipeline } from 'node:stream/promises';

await pipeline(
  fs.createReadStream(inputPath, { highWaterMark: 1 << 20 }),
  new LineTransform(), // stateless: emits per line, never accumulates
  fs.createWriteStream(outputPath)
);

When a loop is unavoidable, respect the write return value. The two-line rule that prevents the crash: if write returns false, stop reading until drain fires.

for await (const chunk of source) {
  if (!out.write(chunk)) {
    await once(out, 'drain');
  }
}

Also search for accidental full-payload buffers: string concatenation in a transform, or a wrapper that does await streamToString(body) before replying.

Verification

Re-run the export with an input at least three times the heap limit, with the interval logger on. Success is threefold: the process completes, heapUsed stays under your ceiling for the whole run, and writableLength oscillates near zero rather than climbing. Add a test that fails if the old write-without-drain pattern returns in review.