Node.js child_process — Error: maxBuffer exceeded

Fix Error: maxBuffer exceeded from exec by measuring real output size, choosing execFile with streams for large output, and parsing incrementally.

beginnerdebuggingnodejschild-processstreams

Symptoms

Error: maxBuffer exceeded at Socket ChildProcess.stdout while running git log, kubectl get, or a database dump command
The same command works in a shell but fails when spawned from Node
Failures started appearing as data grew, with no code change in months

Possible causes

exec buffers stdout in memory and kills the child once it passes the default 1 MB maxBuffer
The command output grows with repository or cluster size, crossing the limit gradually
stderr is redirected into stdout with 2>&1, doubling the buffered volume
Output that should be streamed to a file is instead collected for parsing

Diagnosis steps

Measure the real output size

git log --oneline | wc -c

Interpretation

A byte count above 1048576 explains the failure immediately: the default maxBuffer is 1 MB and the child is killed at the boundary.

Confirm the failing boundary in code

exec(cmd, { maxBuffer: 1024 * 1024 }, cb) # grep -R "maxBuffer" src/

Interpretation

Finding the configured value tells you whether you are at the default or a custom limit someone set earlier.

Decide whether the output must be in memory at all

git log --oneline > /tmp/out.txt && wc -l /tmp/out.txt

Interpretation

If downstream logic only iterates lines, a stream or file beats a bigger buffer in every dimension.

Root cause

child_process.exec collects the entire stdout into a single buffer for convenience and aborts the child process when the buffer exceeds maxBuffer. The output volume grew past the configured ceiling, so a command that once succeeded now dies mid-stream with no partial data delivered.

Fix

For bounded output, set maxBuffer to a justified size with headroom. For anything that scales with data, switch to spawn with readline or pipeline so output is processed line by line, and write bulk output to a file instead of memory.

Risk note

Setting maxBuffer to an enormous value trades a crash for unbounded memory growth under load. Size it from measured output plus headroom, or stream.

Verify the fix

Run the original failing command through the new code path and assert the full output is consumed (line count matches the shell reference) with memory staying flat.

Symptom walkthrough

The error names the mechanism, and the default limit explains the timing:

Error: maxBuffer exceeded
    at Socket.<anonymous> (node:child_process:429:20)

exec exists to give you a callback with the complete output, so it must hold all of it in memory. The default ceiling is 1 MB. Measure the command output yourself:

git log --oneline | wc -c   # e.g. 3_140_000 bytes

Three megabytes against a one-megabyte limit means the child was killed mid-output — which also explains why the same command in your terminal works fine: the shell never buffers a cap.

Fix in practice

If the output is genuinely bounded, raise the limit with justification instead of a magic number:

exec(cmd, { maxBuffer: 16 * 1024 * 1024 }, callback); // measured 3 MB + headroom

If output scales with data, stop buffering it at all and process incrementally:

import { spawn } from 'node:child_process';
import { readline } from 'node:readline/promises';

const child = spawn('git', ['log', '--oneline']);
const rl = readline.createInterface({ input: child.stdout });
let count = 0;
for await (const line of rl) {
  count += 1; // one line in memory at a time
}

For bulk artifacts like dumps, stream stdout straight into a file with pipeline and parse the file afterward. Also check for 2>&1 redirections that silently double what exec has to buffer.

Verification

Compare against the shell as ground truth: pipe the command through wc -l in a terminal, then run the Node path and assert the same count. Watch process.memoryUsage().heapUsed during the run — the streaming variant stays flat regardless of output size, while the buffered variant tracks output volume. Re-run with output twice as large to confirm the streaming path is size-independent.