Stream a Large CSV Export

Write rows to the response as they are read, flush periodically, and backpressure the database cursor so memory stays flat for large exports.

~40 minstreamingdata-exportperformance

Task

Export a large result set as CSV without buffering the whole dataset in memory.

When to use this

An export endpoint holds every row in memory and times out or crashes when the dataset grows, while the user waits with no feedback.

Prerequisites

A database client that supports cursors or streams
A streaming-capable server response

Steps

1. Set the CSV content type and a filename disposition header before writing any rows.

Expected output: The browser starts a download instead of rendering the text.

2. Read rows from a cursor and write each as a CSV line, flushing in batches.

Expected output: Rows leave the server as they are produced.

3. Escape fields containing commas, quotes, or newlines according to RFC 4180.

curl -OJ https://api.example.com/export/users.csv

Expected output: Round-tripping the file reproduces the original values.

4. Handle client disconnects by destroying the cursor to stop the query.

Expected output: Abandoned exports do not keep consuming database resources.

Troubleshooting

Problem

Memory still grows

Solution

Confirm you use a server-side cursor rather than loading all rows into an array before streaming.

Problem

Corrupt rows

Solution

Use a CSV library or a strict quoting function; never join fields with a bare comma.

Limitations

Streaming exports cannot report a total row count or resume after failure without extra bookkeeping.
Very wide rows and slow clients can make the export longer than the query itself.

Context

Export a large result set as CSV without buffering the whole dataset in memory. An export endpoint holds every row in memory and times out or crashes when the dataset grows, while the user waits with no feedback.

Steps

Set the CSV content type and a filename disposition header before writing any rows. Expected: The browser starts a download instead of rendering the text.
Read rows from a cursor and write each as a CSV line, flushing in batches. Expected: Rows leave the server as they are produced.
Escape fields containing commas, quotes, or newlines according to RFC 4180. Expected: Round-tripping the file reproduces the original values.
Handle client disconnects by destroying the cursor to stop the query. Expected: Abandoned exports do not keep consuming database resources.

res.setHeader('Content-Type', 'text/csv; charset=utf-8');
res.setHeader('Content-Disposition', 'attachment; filename="export.csv"');
for await (const row of cursor) res.write(toCsv(row));
res.end();

Verify

The browser starts a download instead of rendering the text.
Rows leave the server as they are produced.
Round-tripping the file reproduces the original values.
Abandoned exports do not keep consuming database resources.

Limits

Streaming exports cannot report a total row count or resume after failure without extra bookkeeping.
Very wide rows and slow clients can make the export longer than the query itself.