阅读中文版本

Build an Image Thumbnail Generation Service

A complete thumbnail service: upload validation, async sharp processing, multiple sizes, content-addressed storage, and malformed-input tests.

thumbnail-service/
├── src/upload.ts
├── src/worker.ts
├── src/variants.ts
├── src/storage.ts
├── tests/thumbnail.test.ts
└── package.json

if (!['image/jpeg','image/png','image/webp'].includes(file.mimetype)) throw new Error('unsupported');

const key = 'orig/' + sha256(buffer) + ext;

for (const w of [200, 600, 1200]) await sharp(input).resize({ width: w, withoutEnlargement: true }).toFile(path(w));

await sharp(input).webp({ quality: 80 }).toBuffer();

await sharp(input).rotate().withMetadata({ orientation: 1 }).toBuffer();

Project overview

Uploading an image and generating thumbnails is a small feature with a long list of ways to be exploited: a malformed file, a decompression bomb, or a CPU-heavy decode that starves the request path. This build validates first and processes asynchronously.

You will validate the content type and size, store the original under a content hash, enqueue a processing job, generate each configured size on a worker, and write the variants to deterministic keys.

Content addressing is what makes the whole thing idempotent. The same bytes uploaded twice map to the same key, so a retry or a duplicate upload does not create a second set of variants or a second copy of the original.

Running the decode on a worker rather than in the request path is the difference between an upload endpoint that stays responsive under load and one that falls over the first time someone uploads several large images at once.

Each phase adds one property: validation, content addressing, variant generation, format choice, and metadata stripping, so the pipeline is safe and useful at every stage.

It is for teams that accept user uploads and have learned that an image pipeline is a decoder running on hostile input. The service separates validation, storage, and processing so each can be reasoned about and bounded on its own.

It is also a good example of a pipeline where the expensive step must be asynchronous: the upload should return as soon as the bytes are safely stored, and the client should learn about the variants through polling or a notification rather than a long-held connection.

What you will build

The service provides five capabilities, each with a test that inspects the stored variants:

• Upload validation
• Content-addressed key
• Variant generation
• Format choice
• Metadata stripping

Upload validation rejects a malformed or oversized file; content addressing makes storage idempotent; a worker generates every configured size; format choice balances quality and size; and metadata stripping removes identifying details from the output.

The design assumes uploads are hostile and processing is expensive, so validation happens before storage and decode happens off the request path.

Architecture and data model

The upload endpoint validates the content type and size, stores the original under a content hash, and enqueues a processing job. A worker decodes with sharp, produces each configured size, and writes them with deterministic keys.

The upload endpoint validates the declared content type against the actual bytes, enforces a size cap, stores the original under a hash of its content, and enqueues a job keyed by that hash.

A worker decodes the original once and produces each configured size, writing variants to deterministic keys derived from the hash and the size, so a retry overwrites the same object rather than creating a new one.

Format choice decides between the original format and a modern encoding per variant, balancing size against compatibility, and records the choice in the key.

Metadata stripping removes EXIF and location data from the variants, because an image uploaded for a thumbnail often carries information the uploader did not intend to publish.

The hash is computed from the content before any processing, so a re-upload of the same file is recognized without decoding it again; that is the property that makes the pipeline idempotent under retries and duplicates.

The key derivation is deterministic from the content hash and the size, so a partial failure that produced some variants can be resumed without recomputing the ones already written.

Prerequisites and dependencies

You need object storage, a queue for jobs, and an image library with the decoders you intend to support.

• Node.js 20 LTS or later
• S3-compatible object storage
• libvips available for sharp

The dependencies cover the HTTP surface, the queue, the image processing, and the storage client:

• sharp (^0.33.4) — Fast image decoding and resizing
• @aws-sdk/client-s3 (^3.620.0) — Original and variant storage
• express (^4.21.0) — Upload endpoint

pnpm init
pnpm add sharp @aws-sdk/client-s3 express
pnpm add -D typescript tsx vitest

Set the size cap, the maximum decode dimensions, and the variant sizes in configuration, because all three bound the resource a single upload can consume, and a generous default is how one image becomes an outage.

Directory structure

Separate upload validation, the content-addressed key, variant generation, format choice, and metadata stripping so each is testable with a fixture image:

thumbnail-service/
├── src/upload.ts
├── src/worker.ts
├── src/variants.ts
├── src/storage.ts
├── tests/thumbnail.test.ts
└── package.json

Phase 1: Upload validation

Check the declared type and the magic bytes, and cap the size.

if (!['image/jpeg','image/png','image/webp'].includes(file.mimetype)) throw new Error('unsupported');

A declared type alone is not proof of content.

Validate the upload by inspecting the bytes rather than the declared type, enforce the size cap, and reject a file whose extension claims an image but whose content does not. Trusting the extension is how a malformed file reaches a decoder.

Phase 2: Content-addressed key

Name the original by a hash of its bytes.

const key = 'orig/' + sha256(buffer) + ext;

Identical uploads deduplicate and cache forever.

Store the original under a content hash so the same bytes always map to the same key, which makes uploads idempotent and prevents duplicate storage of identical files.

Phase 3: Variant generation

Produce each configured size without upscaling.

for (const w of [200, 600, 1200]) await sharp(input).resize({ width: w, withoutEnlargement: true }).toFile(path(w));

Upscaling wastes space and looks worse than the original.

Generate each configured size on a worker by decoding once and resizing in memory, since decoding the original repeatedly per size multiplies the CPU cost of a single upload.

Phase 4: Format choice

Encode WebP for smaller payloads and keep a JPEG fallback.

await sharp(input).webp({ quality: 80 }).toBuffer();

Format negotiation reduces bytes without visible loss.

Choose the output format deliberately, recording the choice in the key, so a variant that is a modern encoding is addressable separately from the original format.

Phase 5: Metadata stripping

Remove EXIF including GPS before publishing.

await sharp(input).rotate().withMetadata({ orientation: 1 }).toBuffer();

Uploaded photos often carry location metadata.

Strip metadata from the variants, because a thumbnail with embedded location data can reveal more than the image itself.

Testing

Upload a real JPEG and assert all variants exist at the right dimensions, then upload a malformed file with an image extension and assert it is rejected.

Upload a real image and assert that every configured variant exists at the expected dimensions, which verifies the worker and the key derivation together.

Upload a malformed file with an image extension and assert it is rejected, because this is the case that a validator using the extension alone would pass straight to the decoder.

Assert that uploading identical bytes twice produces the same key and does not create a second set of variants, which is the idempotency property content addressing exists for.

Assert that the variants contain no metadata, since a stripped field is invisible in a visual check but readable from the file.

Use a small fixture image with a known size so the assertions are about dimensions and keys rather than about pixels, which keeps the suite fast and the failures readable.

Assert that regenerating variants for an existing original overwrites the same keys rather than creating new ones, since a drifting key turns every regeneration into unbounded storage growth.

pnpm vitest run

Deployment and operations

Run processing on workers with a CPU budget; image decode is CPU heavy and can starve request handling if co-located. Configure an object lifecycle rule for orphaned originals.

Run processing on workers with a CPU budget, because an image decode is CPU heavy and will starve request handling if it shares the same process.

Configure an object lifecycle rule for originals that never produced variants, since a failed or abandoned job otherwise leaves an orphan that is paid for indefinitely.

Monitor queue depth and decode duration, because a growing queue means the worker pool is undersized for the upload rate.

Cap concurrent decodes per worker, because memory scales with the decoded dimensions and several large images at once can exhaust a worker.

Give the worker pool a separate CPU budget from the request path, and treat the decode limit as a hard ceiling, because the failure mode of an image pipeline is a burst of large decodes that starves everything else on the host.

Alert on the queue depth rather than on the upload rate, because the queue is what reveals a worker pool undersized for the actual decode cost.

Security considerations

Limit dimensions and total pixels to prevent decompression bombs, strip metadata, and serve variants from a distinct origin or with a restrictive content type.

Inspect the bytes to determine the type and cap the decoded dimensions, because a decompression bomb is a small file that expands to exhaust memory.

Strip metadata and never serve the original under a guessable path, since uploads routinely contain location and device information.

Reject a file whose content type is not on an allowlist, because an image pipeline that accepts arbitrary formats is a decoder vulnerability surface.

Keep uploads and variants private until they are intentionally published, since an unlisted object is still reachable if the storage is public.

Reject an image whose declared dimensions exceed the cap before decoding it, because the dimensions are in the header and a decode of an enormous canvas is exactly the cost the cap exists to avoid.

Sanitize the stored filename or ignore it entirely, because a filename derived from user input is a path traversal risk if it is ever used in a storage path.

Observability

Track processing latency, decode failures, and storage growth. Alert on a rising decode failure rate, which signals a client bug or abuse.

Track upload rate, rejection rate, and queue depth, because together they describe whether the pipeline is keeping up or simply refusing more.

Chart decode duration by input size, since a rising duration for a stable size indicates a decoder or a hardware regression.

Measure the orphan rate, because a growing set of originals without variants means jobs are failing silently.

Alert on rejection spikes, as they usually indicate either an attack or a client sending a wrong format.

Track the ratio of uploads to jobs completed, because a widening gap means jobs are failing or queueing, and both are invisible in an upload success rate.

Track the average input size as well, because a gradual rise means the cap will begin to bind and uploads will start being rejected for a reason nobody intended.

Alert on the orphan rate crossing a threshold, because it is the metric that reveals a decode failure before a user asks where their thumbnail went.

Troubleshooting

Memory spikes on large images. sharp was given a limit far above the pixel cap; enforce max input dimensions.

Thumbnails are rotated wrongly. EXIF orientation was ignored; apply rotate() before resizing and strip the tag.

If variants are missing for one upload, check the job state and the worker log, because a decode failure is recorded on the job rather than failing the upload.

If the same upload produces a different key each time, check the hashing input, because including a timestamp or a filename turns content addressing into random addressing.

If a valid image is rejected, check the byte inspection against the content type, because a validator that reads only the header can reject a legitimate file with an unusual but valid structure.

If a variant is the wrong size, check the configuration and the key, because a variant written under a key from the old size list will be served as though it were the new size.

Limitations and next steps

• No animated image handling.
• No focal-point or smart cropping.
• Sharp runs natively and needs a compatible platform.

The service generates a fixed set of sizes and formats rather than arbitrary transforms, which keeps the storage deterministic but limits flexibility.

There is no synchronous path for a small image, so the client polls or waits for a notification after the variants are ready.

Processing is single-pass per variant family, so an animated image is treated as a still or rejected depending on the configured policy.

There is no automatic re-generation when the size configuration changes; existing variants keep their keys until they are regenerated explicitly.

The service does not implement on-the-fly transforms, so a new size requires a regeneration pass over existing originals rather than a request-time resize.

The pipeline also does not deduplicate variants across formats, so the same visual content stored in two encodings is retained twice.