Go: bufio.Scanner: token too long

Fix bufio.Scanner "token too long" by enlarging the buffer, splitting lines manually, or switching to bufio.Reader for unbounded input.

intermediatedebugginggoio

Symptoms

bufio.Scanner: token too long
Reading a large line or JSON payload fails after ~64KB
Scan returns false and Err() reports the token error

Possible causes

A single line or record exceeds the default 64KB scanner buffer
ScanLines is used on binary or minified data with no newlines for long stretches
The buffer was enlarged by a fixed amount smaller than the actual record
A file was generated without line breaks, forming one huge token

Diagnosis steps

Check scanner.Err() after the loop

Interpretation

Scan returning false with Err()==ErrTooLong confirms the buffer cap was hit rather than a clean EOF.

Measure the longest record in the input

awk "{ if (length > max) max=length } END { print max }" data.txt

Interpretation

A value above the buffer size tells you how much headroom is needed.

Decide between larger buffer and Reader

Interpretation

If records are genuinely unbounded, a fixed buffer of any size will eventually fail; Reader avoids the cap.

Root cause

bufio.Scanner reads tokens into a fixed-size buffer that defaults to 64KB. If a single token (line by default) is larger than the buffer, Scan returns false and Err() reports ErrTooLong. The scanner cannot grow the buffer itself, so it is the caller's responsibility to size it or switch APIs.

Fix

Either raise the limit with scanner.Buffer(make([]byte, 0, max), max) sizing to the real maximum record, or use bufio.Reader with ReadString/ReadBytes so memory grows with the input. For structured data, prefer a streaming decoder like json.Decoder over line-oriented scanning.

Verify the fix

Re-scan the previously failing input; the loop reaches EOF with scanner.Err() returning nil and every record is processed, confirming the buffer now matches the data.

Symptoms

Scan stops early and scanner.Err() returns bufio.Scanner: token too long. It typically fails on a single oversized line while shorter lines work fine, and the default cap is 64KB.

Diagnosis

Always check the scanner error and measure the longest record:

sc := bufio.NewScanner(f)
for sc.Scan() {
    process(sc.Text())
}
if err := sc.Err(); err != nil {
    log.Fatalf("scan failed: %v", err)
}

A longest record above 64KB explains the failure.

Fix

Size the buffer for a known maximum, or use a Reader for unbounded input:

const maxCap = 10 * 1024 * 1024
sc := bufio.NewScanner(f)
sc.Buffer(make([]byte, 0, 64*1024), maxCap)

// or stream without a cap
r := bufio.NewReader(f)
line, err := r.ReadString('\n')

Verification

Re-run on the failing input; the loop reads to EOF and sc.Err() is nil, proving the buffer accommodates the data.

Additional Notes

The scanner cap is a convenience default, not a protocol limit, so raising it is usually right for line-oriented data while reorganising truly unbounded records is right for machine-generated payloads. Logging the maximum observed record length before sizing avoids guesswork, and reserving a little headroom above that maximum keeps occasional larger lines from failing later. When the input is structured, decoding it with a streaming decoder is more robust than treating it as lines, because records then follow their own framing instead of an arbitrary newline assumption.