阅读中文版本

Build an Asynchronous Data Export Service

A complete export service: export jobs, streaming CSV generation, private signed downloads, expiry, progress reporting, and large-dataset tests.

data-export/
├── src/enqueue.ts
├── src/worker.ts
├── src/csv-stream.ts
├── src/download.ts
├── tests/export.test.ts
└── package.json

{ id, filter, requestedBy, status: 'queued', progress: 0 }

const q = new QueryStream('SELECT * FROM orders WHERE ...');
await pipeline(q, new CsvTransform(columns), upload);

if (written % 1000 === 0) await db.setProgress(jobId, written / total);

const url = await s3.getSignedUrl('getObject', { Bucket, Key, Expires: 3600 });

await s3.deleteObject({ Bucket, Key }); await db.markExpired(jobId);

Project overview

Exports are the feature nobody plans for and everybody needs: a report that must be complete, private, and finished before anyone is watching a spinner. This build generates large exports in the background and hands back a link that expires.

You will create an export job and return its id, stream the rows through a CSV transform into object storage without holding them all in memory, track progress, and issue a short-lived signed URL when the job completes.

The two constraints that shape everything are memory and privacy. Streaming keeps a million-row export from becoming a million-row heap, and a signed, expiring link keeps a private dataset from becoming a public download.

The job model is what makes the work observable: a request returns immediately, the client polls or is notified, and the operator can see which exports are running and which failed.

The phases build outward from the job record: streaming, progress, secure delivery, and cleanup, each one meaningfully usable on its own.

The export is where a dataset leaves the system that governs it, so every decision here is as much about governance as about engineering. The job record, the private artifact, and the expiry sweep exist to keep that exit deliberate.

What you will build

The service provides five capabilities, each with a test that runs a real export:

• Export job
• Streaming CSV
• Progress
• Signed download
• Expiry cleanup

A job record decouples the request from the work; streaming writes rows without buffering; progress makes a long job visible; a signed download keeps the artifact private; and an expiry sweep removes data that is no longer needed.

The export runs on its own worker pool so a large report never competes with request handling. That separation is the difference between a background job and a self-inflicted outage.

Architecture and data model

An export request creates a job row and returns a job id. A worker streams a cursor of rows through a CSV transform into object storage, updating progress. Completion stores a private key and expiry; the download endpoint issues a short-lived signed URL.

An export request inserts a job row with a status and returns the id. A worker claims the job, streams a cursor of rows through a CSV transform into object storage, and updates progress as it goes. Completion records a private key and an expiry.

The download endpoint issues a signed URL with a short lifetime, so the artifact is never public and the link cannot be shared indefinitely. The object itself stays private; only the signature grants access.

Progress is stored on the job row so the client can poll without holding a connection open. That keeps the export resilient to a client that navigates away or a network that drops.

Because the worker uses a server-side cursor rather than a single result set, memory stays flat regardless of row count. That is the property a naive implementation loses first, and it is why the streaming design is not optional.

The job row is the single source of truth for an export, carrying status, progress, artifact key, and expiry. Keeping all of that in one row means a client, an operator, and the cleanup job all read the same state instead of three.

Prerequisites and dependencies

You need a database to read from and object storage to write artifacts to, plus a way to sign URLs.

• Node.js 20 LTS or later
• PostgreSQL 15+
• S3-compatible object storage

The dependencies cover job storage, streaming, and signed object access:

• pg (^8.12.0) — Cursor-based row streaming
• @aws-sdk/client-s3 (^3.620.0) — Upload export files
• express (^4.21.0) — Enqueue and download endpoints

pnpm init
pnpm add pg @aws-sdk/client-s3 express
pnpm add -D typescript tsx vitest

Keep the storage credentials and the signing key in a secret manager, and validate the bucket name at boot. Publish the signed URL lifetime as configuration so a security review can tighten it without a code change.

Directory structure

Separate job creation, the streaming writer, progress updates, signed delivery, and the expiry sweep so each concern is testable:

data-export/
├── src/enqueue.ts
├── src/worker.ts
├── src/csv-stream.ts
├── src/download.ts
├── tests/export.test.ts
└── package.json

Phase 1: Export job

Persist the filter, requester, status, and progress.

{ id, filter, requestedBy, status: 'queued', progress: 0 }

A job record makes long exports observable and cancellable.

Create the job first and return its id, so the request completes before any rows are read. A synchronous export couples the client timeout to the dataset size and fails exactly when the data is most interesting.

Phase 2: Streaming CSV

Transform each row to a CSV line and pipe to the upload.

const q = new QueryStream('SELECT * FROM orders WHERE ...');
await pipeline(q, new CsvTransform(columns), upload);

Streaming keeps memory flat regardless of row count.

Stream the rows through the transform on the worker, flushing as you go, so peak memory is independent of row count. A test that exports more rows than one batch is the only convincing proof of this.

Phase 3: Progress

Update progress periodically by rows written over estimated total.

if (written % 1000 === 0) await db.setProgress(jobId, written / total);

Operators and users need to know an export is progressing.

Update progress on the job row periodically rather than per row, so a long export is observable without turning progress writes into a bottleneck. Progress is for humans; it should not dominate the write load.

Phase 4: Signed download

Store a private object and hand out a short-lived signed URL.

const url = await s3.getSignedUrl('getObject', { Bucket, Key, Expires: 3600 });

A permanent public URL would leak exported data.

On completion, store the object under a private key and mint a signed URL with a short lifetime. The signature is the capability, so its expiry is the real access control.

Phase 5: Expiry cleanup

Delete export files and mark jobs expired after the retention window.

await s3.deleteObject({ Bucket, Key }); await db.markExpired(jobId);

Exports contain bulk data and must not live forever.

Expire artifacts on a schedule and delete the object when the job ages out. An export that lives forever is an unmanaged copy of your data, which is the privacy problem this step exists to avoid.

Testing

Export a dataset large enough to exceed one batch and assert the row count matches, memory stays bounded, and an expired job denies download.

Export a dataset large enough to exceed one batch and assert the row count matches the source, then assert that peak memory stays bounded, since a passing row count with a runaway heap is still a production failure.

Assert that an expired job denies download and that a tampered signature returns 403. Both are access-control tests, and both are cheap to write with a fixed clock.

Test that a failed export records the error on the job rather than leaving it running forever, so an operator can see and retry it.

Check CSV escaping with a fixture containing commas, quotes, and newlines, because an export that corrupts a field is worse than one that fails loudly.

Add a test that cancels a running export and asserts the worker stops and no artifact is published, because cancellation is easy to forget and leaves orphans in storage.

pnpm vitest run

Deployment and operations

Run exports on dedicated workers so a large job cannot starve request handling. Set object lifecycle rules as a backstop for cleanup.

Run exports on dedicated workers so a large job cannot starve request handling, and set object lifecycle rules as a backstop in case the cleanup sweep fails. Two independent expiry mechanisms are worth the small duplication.

Size the worker pool to the storage and database read budget rather than to the number of pending jobs, because parallel exports multiply read load in a way that is invisible per job.

Store artifacts in a bucket with no public access, and sign URLs from a key kept separate from other application secrets so it can be rotated independently.

Emit a job id in every log line and hand it to the client, so a support conversation can be joined to the exact export attempt rather than to a vague timestamp.

Decide the artifact retention period per dataset sensitivity rather than globally, and record that decision with the job so the cleanup sweep can apply the right period without a guess.

Give export workers a read replica where one is available, so a heavy report does not compete with transactional traffic for connections on the primary.

Security considerations

Scope downloads to the requester and expire links quickly. Redact sensitive columns by policy and audit who exported what and when.

Treat the download link as a bearer capability: keep it short-lived, keep the object private, and never log the full URL, because a signed URL in a log is a credential in a log.

Redact or exclude columns a requester is not entitled to see, and enforce that at query time rather than trusting the client to ignore extra fields.

Limit concurrent exports per account so a single user cannot exhaust the worker pool, which is a denial-of-service vector disguised as a feature.

Delete artifacts on expiry and verify the deletion, because the privacy promise depends on the data actually leaving the bucket, not merely on the link expiring.

Bind the signed URL to the requesting identity where possible, or at least keep its lifetime short enough that a leaked link is a small window, and always serve it over HTTPS.

Never place the artifact key in the URL path without a signature, and keep bucket listing disabled, so the only way to reach an artifact is a link you minted.

Observability

Track job duration, rows per second, failure rate, and storage used. Alert on jobs stuck at the same progress, which indicates a stalled stream.

Track exports started, completed, and failed, with durations per size bucket. A rising duration for a stable size points at a database or storage regression rather than at the export code.

Chart worker pool saturation and queue wait, because a growing wait is the earliest sign that exports need more capacity before any client notices a delay.

Alert on cleanup failures, since a broken sweep quietly accumulates private copies of data, which is a compliance problem that does not announce itself.

Record the size and row count of each artifact so a sudden spike is visible as data drift rather than discovered later as a storage bill.

Track time from job completion to first download, because an artifact that is never downloaded is pure cost, and that signal is useful for pruning export features nobody uses.

Track artifact bytes stored per day, because a rising trend at a flat job count means exports are getting larger and the retention policy needs review before storage costs surprise you.

Troubleshooting

Worker runs out of memory. The whole result set was buffered; stream with a cursor and backpressure.

CSV is malformed for values with commas. Values were not quoted or escaped; use a correct CSV encoder.

If memory climbs during an export, the worker is buffering rows; check that the cursor is being streamed and not collected into an array before the transform.

If a completed export returns 403, check the URL expiry before the signature, since an expired short-lived link is the intended behavior rather than a bug.

If an export finishes but the client never sees a link, check the job status and the artifact key together; a completed job with a missing key means the upload half-finished, and the cleanup sweep should treat that as a failure rather than a success.

Limitations and next steps

• CSV only; no Parquet or XLSX writer.
• No resumable downloads.
• Column redaction is a fixed policy, not per-request.

Exports are single-pass and sequential; there is no parallel range read, so very large datasets are bounded by one worker’s throughput.

The artifact format is CSV only and the column set is fixed at request time rather than user-defined, which keeps the transform simple at the cost of flexibility.

There is no push notification on completion; clients poll the job status. Adding a webhook or an email on completion is a small extension the job model already supports.

Cross-export caching is absent, so exporting the same view twice does the work twice. A content-addressed cache keyed by query and version would remove that duplication.

The service exports a fixed query shape and does not expose an arbitrary query interface, which is a deliberate restriction rather than a missing feature.

Progress is coarse and based on rows processed rather than bytes written, so a client polling progress sees a step function rather than a smooth bar.