Transform CSV to NDJSON in a Stream

Convert a large CSV to newline-delimited JSON with a streaming reader and writer so memory stays flat regardless of file size.

~25 mindatapythonetl

Task

Stream-convert CSV to NDJSON to keep memory constant for large inputs.

When to use this

A batch import expects NDJSON, but loading the CSV into memory to convert it exhausts the worker.

Prerequisites

Python 3 with csv and json
A CSV input and NDJSON output path
Enough disk for the output

Steps

1. Stream rows and write one JSON object per line.

import csv, json
with open('in.csv', newline='') as fin, open('out.ndjson', 'w') as fout:
    for row in csv.DictReader(fin):
        fout.write(json.dumps(row, ensure_ascii=False) + '\n')

Expected output: Each row becomes one JSON line, written as it is read.

2. Spot-check the output for correctness.

head -1 out.ndjson | python -m json.tool
wc -l in.csv out.ndjson

Expected output: The first line is valid JSON and line counts match (minus the header).

Troubleshooting

Problem

Types are all strings

Solution

Cast known numeric columns before serialising so consumers get real numbers.

Problem

Encoding garbled

Solution

Open both files with an explicit encoding such as utf-8.

Limitations

Streaming cannot sort or aggregate across rows.
Type inference needs explicit rules.
No schema enforcement by itself.

What this recipe does

Stream-convert CSV to NDJSON to keep memory constant for large inputs.

A batch import expects NDJSON, but loading the CSV into memory to convert it exhausts the worker.

Prerequisites

Python 3 with csv and json
A CSV input and NDJSON output path
Enough disk for the output

Procedure

Stream rows and write one JSON object per line.

import csv, json
with open('in.csv', newline='') as fin, open('out.ndjson', 'w') as fout:
    for row in csv.DictReader(fin):
        fout.write(json.dumps(row, ensure_ascii=False) + '\n')

Expected result: Each row becomes one JSON line, written as it is read.

Spot-check the output for correctness.

head -1 out.ndjson | python -m json.tool
wc -l in.csv out.ndjson

Expected result: The first line is valid JSON and line counts match (minus the header).

How to confirm it worked

Run the commands above and confirm every expected result in order. If a step does not produce the stated output, use the troubleshooting notes before continuing. A passing run shows the output has one valid JSON object per line, the counts line up, and memory stays flat for a large input.

Troubleshooting

Types are all strings — Cast known numeric columns before serialising so consumers get real numbers.

Encoding garbled — Open both files with an explicit encoding such as utf-8.

Limitations

Streaming cannot sort or aggregate across rows.
Type inference needs explicit rules.
No schema enforcement by itself.