阅读中文版本

Build a File Watch Pipeline

A pipeline that watches a directory for changes, debounces rapid saves, runs a transform on each changed file, and emits results with a manifest of what was processed.

file-watch-pipeline/
├── src/watcher.ts
├── src/debounce.ts
├── src/dispatcher.ts
├── tests/pipeline.test.ts
└── package.json

const w = fs.watch(dir, {recursive:true}); w.on("change", (e, f) => emit(join(dir,f)));

timers.set(file, setTimeout(() => emit(file), 200));

try { await transforms[ext](file); } catch (e) { manifest.fail(file, e); }

manifest.add({file, ok:true, ts:Date.now()}); await manifest.flush();

Project overview

A build step that runs on every save is the development loop, but a build that runs on every partial save is noise. An editor that saves five times while you type a line should trigger one transform, not five. A file watch pipeline debounces rapid saves into one event per file, so the transform runs once when you stop typing.

The recursive watch covers the directory and all subdirectories, so a change in a nested file triggers the pipeline. The watcher wraps the platform fs.watch, which is efficient because it uses the OS file events, not polling. The trade-off is platform dependence: the event names and the recursive support vary, so the watcher abstracts the differences.

The transform per file type lets the pipeline handle different files differently. A markdown file runs a markdown transform, a TypeScript file runs a type check, and an image file runs a compression. The dispatcher picks the transform by extension, so adding a file type is adding a transform, not changing the pipeline.

The manifest is what makes the pipeline observable. Every processed file is recorded with its result and timestamp, so an operator can see what the pipeline did and when. The manifest is written atomically, so a crash leaves a valid file, not a truncated half-entry.

The value of this project is the debounce, not the watch. Watching a directory is trivial; collapsing rapid saves into one transform is the part worth learning, because that is what makes a file watch pipeline usable in a development loop instead of a noise generator.

What you will build

The project delivers five capabilities, each driven by a test:

• Recursive directory watch
• Debounce rapid saves
• Transform per file type
• Manifest of processed files
• Error recovery without restart

A watcher wraps fs.watch recursively, emits change events, and feeds a debouncer that collapses rapid saves into one event per file. A dispatcher runs the transform for the file type, catches errors per file, and appends to a manifest. The manifest is written atomically so a crash leaves a valid file.

Architecture and data model

The watcher wraps fs.watch with the recursive option. The recursive option is supported on most modern systems and falls back to a manual directory walk on systems that do not support it. The watcher emits a change event with the full file path, normalizing the event type to a single “change” to abstract platform differences.

The debouncer holds a map of file path to a timer. On a change event, it clears the existing timer for that file and sets a new one. When the timer fires, the file is emitted as a debounced change. The window is a tunable, typically 200 milliseconds, which is long enough to collapse editor saves and short enough to feel responsive.

The dispatcher receives debounced changes and runs the transform for the file type. The transform is a function from file path to a result, selected by extension from a map. The dispatcher catches errors per file, so a failing transform does not stop the pipeline. The error is recorded in the manifest, and the dispatcher proceeds to the next file.

The manifest is an array of entries, each a file path, a result, and a timestamp. The manifest is flushed to disk atomically: the new content is written to a temporary file, then renamed over the manifest. A crash during the flush leaves the old manifest, because the rename is the last step, so the manifest is always valid.

The error recovery is the per-file try-catch. A transform that throws does not stop the watcher or the dispatcher; it is recorded in the manifest and the pipeline continues. A transform that throws on every save for the same file is a recurring failure, and the manifest shows it, so an operator can fix the transform without restarting the pipeline.

Prerequisites and setup

You need the following before you begin:

• Node.js 20 LTS or later
• A directory to watch
• A transform function per file type

The dependencies cover the core needs:

• node:fs (built-in) — Recursive watch and file read
• node:path (built-in) — File type detection

pnpm init

pnpm add -D typescript tsx vitest

Keep secrets and configuration in the environment, and validate them at boot so a misconfigured service fails loudly instead of silently dropping work.

Directory structure

file-watch-pipeline/ is organized so each concern is testable in isolation:

file-watch-pipeline/
├── src/watcher.ts
├── src/debounce.ts
├── src/dispatcher.ts
├── tests/pipeline.test.ts
└── package.json

Phase 1: Recursive watch

Watch the directory and all subdirectories, emitting change events with the file path.

const w = fs.watch(dir, {recursive:true}); w.on("change", (e, f) => emit(join(dir,f)));

Recursive watch is platform-supported on most systems and falls back to a manual walk.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 2: Debounce

Collapse rapid saves from an editor into one event per file within a window.

timers.set(file, setTimeout(() => emit(file), 200));

An editor that saves five times in 100ms produces one transform, not five.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 3: Dispatch and transform

Run the transform for the file type, catching errors per file.

try { await transforms[ext](file); } catch (e) { manifest.fail(file, e); }

A failing transform does not stop the pipeline or skip the next file.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 4: Manifest

Append each processed file and its result to a manifest, written atomically.

manifest.add({file, ok:true, ts:Date.now()}); await manifest.flush();

A crash leaves a valid manifest because the flush is atomic.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Testing

Touch a file and assert the transform runs once; save five times rapidly and assert the transform runs once; throw in a transform and assert the manifest records the failure and the next file still processes; restart and assert the manifest is valid.

The single-change test touches a file and asserts the transform ran once. The debounce test saves the file five times within the debounce window and asserts the transform ran once, proving the debouncer collapsed the saves.

The error-recovery test throws in a transform for one file, then changes a second file, and asserts the second transform ran and the manifest recorded the first failure. The manifest-validity test kills the pipeline during a flush and asserts the manifest is valid, proving the atomic write.

The recursive test creates a subdirectory, touches a file in it, and asserts the transform ran, proving the recursive watch. The transform-selection test changes a markdown file and a TypeScript file and asserts each ran its own transform, proving the dispatch by extension.

The restart test starts the pipeline, processes some files, stops it, restarts it, and asserts the manifest is read and new changes are processed. The no-duplicate test saves a file, waits for the transform, saves again, and asserts the transform ran twice, once per debounced save, not more.

pnpm vitest run

Observability

Track the recursive directory watch rate, the file-watch resource usage, and the error count as the three primary signals for the file-watch-pipeline system. A sudden change in any of these is the signal that something has shifted, not necessarily broken but worth a look before it becomes an incident. Chart the debounce rapid saves alongside the recursive directory watch so a correlation is visible at a glance, because a rising debounce rapid saves with a stable recursive directory watch means the system is doing more work per unit, not that it is failing.

Log every recursive watch transition with the context that triggered it, because an operator who sees the symptom but not the cause will guess wrong. The log entry should carry the file-watch identifier, the input that caused the transition, and the outcome, so the operator can reconstruct what happened without re-running it. A log without context is a line of text; a log with context is a diagnosis.

Set alerts on the derived metrics, not the raw counters. A rate of change is more actionable than a total, because a total grows monotonically and never tells you when the growth accelerated. Alert on the recursive directory watch error rate crossing a threshold, the file-watch utilization exceeding a percentage, and the debounce rapid saves latency exceeding the SLO. Each alert should have a runbook link, so the operator who receives it at three in the morning knows what to do, not just what is wrong.

The file-watch-pipeline system should expose its metrics in a format the existing dashboard can scrape, so the observability is additive, not a new tool. A metric that nobody reads is noise, so before adding one, confirm it has a consumer. The dashboard has three panels: throughput, error rate, and resource utilization, each with a threshold line that is the SLO, so a glance tells an operator whether the system is healthy without reading numbers.

Deployment and operations

Run the pipeline as a long-lived process, typically started at the beginning of a development session and stopped at the end. Point it at the source directory and the output directory, and let it run. The pipeline is the development loop, so it should run for as long as the developer is working.

Choose the debounce window from the editor save behavior. An editor that saves on every keystroke needs a longer window; an editor that saves on blur needs a shorter one. The window is a tunable, and the right value is the one that feels responsive without running on partial saves.

Monitor the manifest for failures. A file that fails on every save is a recurring error, and the manifest shows it as a sequence of failed entries. The transform should be fixed, not the pipeline restarted, because the pipeline is designed to continue past failures.

Do not use the pipeline for production builds. The pipeline is for the development loop, where speed and continuation matter more than completeness. A production build runs once, on all files, and fails on any error, which is a different mode.

Security considerations

Treat every input to the file-watch-pipeline system as untrusted, because the recursive directory watch path is the one place an external caller controls what the system processes. Validate the input shape before any file-watch operation, and reject early with a clear error rather than letting a malformed input propagate into the recursive watch phase where it causes a confusing failure that is hard to trace back to its source.

Keep secrets and configuration in the environment, never in the repository. The node:fs credentials, the signing keys, and the service endpoints are all environment variables, read once at boot and validated, so a misconfigured file-watch-pipeline fails loudly at startup instead of silently misbehaving at runtime. A system that starts with a bad configuration and runs anyway is a system that will produce wrong results for hours before anyone notices.

Rate-limit the recursive directory watch endpoint per caller, because an unbounded file-watch operation is a resource leak as much as a security problem. A caller that can trigger the debounce path without limit can exhaust the system, whether intentionally or not. The rate limit is a per-caller ceiling that costs legitimate callers nothing and stops the abuse before it reaches the recursive watch logic.

Audit-log every state-changing operation with the caller identity, the input, and the timestamp. The audit log is not for debugging, it is for accountability: if something went wrong, the log says who did what and when. The file-watch-pipeline system writes the audit log to an append-only store that the system itself cannot modify, so a compromised system cannot rewrite its history.

Do not expose internal errors to the caller. A stack trace is a map of the system internals, and an attacker who collects them learns the file-watch structure. Return a generic error with a correlation id, and log the full error internally with the same id, so an operator can trace it without leaking it.

Limitations and next steps

• fs.watch is platform-dependent; some systems report events differently.
• No symlink following; a symlinked directory is not watched.
• The manifest grows; a rotation policy is out of scope.

fs.watch is platform-dependent. Some systems report events with different names, some do not support recursive watch, and some report spurious events. The watcher abstracts the common cases but cannot abstract every platform quirk, so a platform-specific issue may require a platform-specific fix.

There is no symlink following. A symlinked directory is not watched, because following symlinks can create cycles and the watcher does not detect them. A real use case that needs symlink watching should use a watcher that resolves and tracks symlinks, which is a more complex implementation.

The manifest grows without rotation. Every processed file is appended, so the manifest grows for the lifetime of the pipeline. A rotation policy that keeps the manifest bounded is a real improvement but out of scope, because the manifest is for the session, not for history.