阅读中文版本

Build a Time-Series Database Aggregator

Time-Series Database Aggregator — a complete walkthrough of time-series data aggregation, covering downsample buckets, retention policy, compression encoder, and testing.

time_series_db_aggregator/
├── src/index.ts
├── src/downsample-buckets.ts
├── src/retention-policy.ts
├── tests/time_series_db_aggregator.test.ts
└── package.json

// Downsample buckets: time-series data aggregation core
export function downsamplebuckets(input: DownsamplebucketsInput): DownsamplebucketsResult {
  // Build the downsample buckets that anchors time-series data aggregation and defines the core data shape.
  return processDownsamplebuckets(input);
}

// Retention policy — handles time-series data aggregation Add the retention policy so
export class Retentionpolicy {
  constructor(private opts: RetentionpolicyOptions) {}
  async run(ctx: Context): Promise<RetentionpolicyOutput> {
    return this.execute(ctx);
  }
}

// Compression encoder: time-series data aggregation stage
export const compressionencoder = (next: Next) => async (req: Request) => {
  const out = await transform(req);
  return next(out);
};

// Query rollup for time-series data aggregation
export async function queryrollup(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
  let acc = init();
  for await (const c of stream) acc = merge(acc, c);
  return acc;
}

// Ingest backpressure: time-series data aggregation finalization
export function ingestbackpressure(state: State): Result {
  validate(state);
  return finalize(state);
}

Project overview

This project builds a time-series data aggregation system whose reason for existing is compressing high-frequency samples into queryable buckets without losing p99 fidelity. The temptation in time-series data aggregation is to bolt the interesting part onto a working prototype and discover the hard cases in production; this build inverts that order and treats a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption as the first thing designed, not the last.

You will implement Downsample buckets, Retention policy, Compression encoder, Query rollup, and Ingest backpressure, each as a module with one responsibility and a test that proves it in isolation. The time-series data aggregation pipeline is then the wiring that composes them, and the integration test is where the composition is shown to survive a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption.

The walkthrough is written for a developer who has used a time-series data aggregation system but has not had to make one honest. It favors explicit data shapes over clever abstractions, and it names the failure each design choice prevents rather than presenting the choice as a style preference.

Each phase ends with something observable — a downsample buckets that returns the right shape, a retention policy that rejects bad input, an integration test that goes green — so you can stop at any point and still have a time-series data aggregation system that works for the cases covered so far.

What you will build

The finished time-series data aggregation system exposes five capabilities, each with a focused test:

• Downsample buckets
• Retention policy
• Compression encoder
• Query rollup
• Ingest backpressure

Downsample buckets is the foundation that every later stage assumes; Retention policy is where time-series data aggregation correctness lives; Compression encoder handles a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption; Query rollup makes the system observable; and Ingest backpressure is the graceful-degradation layer that keeps overload from becoming corruption.

The build keeps the time-series data aggregation logic in ordinary modules you can read in one sitting, and pulls in dependencies only for the parts that are genuinely tedious to hand-roll. That keeps the diff between the tutorial and a real codebase small, and keeps the failure modes visible in code rather than hidden behind a framework.

Architecture and data model

The system is organized around Downsample buckets, Retention policy, and Compression encoder, with Query rollup tying them together. Data flows as a column of timestamps paired with scalar values, bucketed by a configurable interval and stored as delta-of-delta encoded blocks.

Each component is a module with a pure interface where practical: the same input produces the same output, and side effects are pushed to the edges. That discipline is what makes the phases independently testable, because a test for Compression encoder does not need to stand up Downsample buckets and Retention policy to exercise the interesting branch.

The data model is deliberately explicit. A column of timestamps paired with scalar values, bucketed by a configurable interval and stored as delta-of-delta encoded blocks Keeping the shape written down — in a schema, a type, or a fixture — means a change to the shape is a visible diff rather than a silent runtime shift, and it means the integration test can assert against a concrete shape instead of guessing.

Configuration and code are kept apart: the configuration says what time-series data aggregation should do, the modules say how. A reviewer can change a threshold or a policy without reading the implementation, which keeps the risky changes small and lets the diff stand as the review.

The failure path through Compression encoder is designed before the happy path is finished, because a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption is the case that distinguishes a time-series data aggregation system from a demo. Every component declares what it does when its inputs are wrong, and the integration test asserts each of those declarations.

Prerequisites and dependencies

Have these ready before you install anything, so the time-series data aggregation setup is not blocked by a missing tool:

• Node.js 20 LTS or later
• Familiarity with time-series data aggregation fundamentals
• A local environment to run the test suite

The runtime dependencies are chosen to keep the time-series data aggregation datapath legible:

• express (^4.21.0) — HTTP ingest and query surface
• lmdb (^3.2.0) — Memory-mapped store for buckets
• msgpack (^3.0.0) — Compact encoding for samples

# setup for time-series-db-aggregator
pnpm init
pnpm add express lmdb msgpack
pnpm add -D typescript tsx vitest

Versions are pinned to a major range so the time-series data aggregation install is reproducible; commit the lockfile. The downsample buckets reads its operational parameters from the environment and validates them at startup, so a missing value fails the boot rather than the first time-series data aggregation request.

Directory structure

Keep each time-series data aggregation component in its own file so it can be tested without starting the others:

time_series_db_aggregator/
├── src/index.ts
├── src/downsample-buckets.ts
├── src/retention-policy.ts
├── tests/time_series_db_aggregator.test.ts
└── package.json

Phase 1: Downsample buckets

Build the downsample buckets that anchors time-series data aggregation and defines the core data shape.

// Downsample buckets: time-series data aggregation core
export function downsamplebuckets(input: DownsamplebucketsInput): DownsamplebucketsResult {
  // Build the downsample buckets that anchors time-series data aggregation and defines the core data shape.
  return processDownsamplebuckets(input);
}

After this phase the downsample buckets works in isolation and is covered by a unit test. The downsample buckets defines the data shape that every later stage assumes, so getting it right first means the later phases compile against a stable contract.

Treat the downsample buckets as the single source of truth for the time-series data aggregation data shape. Validate inputs at the boundary and reject early, so a malformed value never reaches Retention policy or Compression encoder where its effect would be harder to localize.

Phase 2: Retention policy

Add the retention policy so time-series data aggregation can compressing high-frequency samples into correctly.

// Retention policy — handles time-series data aggregation Add the retention policy so
export class Retentionpolicy {
  constructor(private opts: RetentionpolicyOptions) {}
  async run(ctx: Context): Promise<RetentionpolicyOutput> {
    return this.execute(ctx);
  }
}

After this phase the retention policy works in isolation and is covered by a unit test. This is where time-series data aggregation correctness lives: the retention policy is what makes compressing high-frequency samples into queryable buckets without losing p99 fidelity true, and its test is the proof.

Keep the retention policy pure where you can. A function that takes an input and returns an output is trivial to test and impossible to mock wrong; a class with hidden state is the opposite. Push side effects to the caller and keep the retention policy about the transformation.

Phase 3: Compression encoder

Wire the compression encoder to handle a write burst that without losing data.

// Compression encoder: time-series data aggregation stage
export const compressionencoder = (next: Next) => async (req: Request) => {
  const out = await transform(req);
  return next(out);
};

After this phase the compression encoder works in isolation and is covered by a unit test. The compression encoder is the difference between a time-series data aggregation system that works on the happy path and one that survives a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption.

Design the failure before the success. Decide what compression encoder does when a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption: reject, retry, shed, or degrade — and write that test first. A time-series data aggregation system whose failure behavior is unspecified will pick one at random under pressure, and it is usually the wrong one.

Phase 4: Query rollup

Integrate the query rollup to make time-series data aggregation observable and testable end to end.

// Query rollup for time-series data aggregation
export async function queryrollup(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
  let acc = init();
  for await (const c of stream) acc = merge(acc, c);
  return acc;
}

After this phase the query rollup works in isolation and is covered by a unit test. Observability is not a finishing touch; it is how you know the time-series data aggregation system is doing what it claims in production.

Emit one structured event per meaningful action, with enough context to localize a failure without a second round-trip. The query rollup is what turns “something is slow” into “the retention policy is slow for inputs matching this shape,” which is the difference between an alert and an investigation.

Phase 5: Ingest backpressure

Finalize the ingest backpressure so the time-series data aggregation system degrades gracefully under overload.

// Ingest backpressure: time-series data aggregation finalization
export function ingestbackpressure(state: State): Result {
  validate(state);
  return finalize(state);
}

After this phase the ingest backpressure works in isolation and is covered by a unit test. The ingest backpressure is what keeps overload from becoming corruption: when the time-series data aggregation system is asked for more than it can do, it degrades in a defined way instead of failing in an undefined one.

Pick a degradation strategy for ingest backpressure — shed, queue, or backpressure — and make it explicit in code and in a test. Leaving the behavior to chance is how a time-series data aggregation traffic spike becomes a data loss incident.

Testing

Each phase has a focused test: downsample buckets, retention policy, and compression encoder are exercised in isolation, then an integration test drives the full time-series data aggregation pipeline and asserts the a write burst that case is handled.

Start with unit tests that pin each component’s contract: the downsample buckets returns the documented shape, the retention policy transforms correctly, and the compression encoder fails in the documented way. These tests are fast and they localize regressions to the component that broke.

Then add an integration test that composes the components and drives a realistic time-series data aggregation scenario end to end. Assert the happy path first, then deliberately trigger a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption and assert the system degrades rather than corrupts. Because the components are pure where practical, the integration test stays deterministic rather than flaky.

A time-series data aggregation suite must also prove the negative: bad input is rejected at the boundary, overload is shed rather than queued indefinitely, and a partial failure does not leave the system in an inconsistent state. These are the tests that catch the regressions a happy-path suite never would.

The entire time-series data aggregation suite runs with no external service; every dependency is either in-process or stubbed, so every case is reproducible in CI. That is what lets the compression encoder failure-mode assertions live as tests rather than as a manual checklist that drifts out of date.

# run the time-series data aggregation test suite
pnpm vitest run

Deployment and operations

Run at least two time-series data aggregation instances behind a load balancer and share state in Redis. Set resource limits so time-series data aggregation overload surfaces as backpressure rather than failure.

Run at least two stateless instances behind a load balancer, and move any shared state into a shared store so the time-series data aggregation behavior is global rather than per instance. Size limits so the system sheds load before it runs out of memory, because a time-series data aggregation system that OOMs under load fails in a way that is hard to reason about.

Drain on deploy: stop accepting new work, let in-flight time-series data aggregation operations finish, then exit. Because the system holds no state that cannot be reconstructed, a rollout is a matter of cycling instances one at a time while watching the error rate.

When a dependency is unavailable, decide explicitly whether the time-series data aggregation system fails closed or degrades. Leaving that choice to chance is how a dependency outage becomes a silent data loss, so make the choice visible in code and in a test.

Security considerations

Validate all time-series data aggregation inputs at the boundary, reject oversized payloads before buffering them, and keep secrets in the environment rather than the repository.

The time-series data aggregation system is a choke point, so it is the right place to reject malformed input before it reaches retention policy or compression encoder. Validate the shape and the size at the edge, and let the inner components assume what they receive is well-formed.

Never log secrets or full payloads; log enough to localize a failure without recording data that should not be on disk. Treat any external call as untrusted and validate its response before using it, because a time-series data aggregation system that trusts its inputs blindly is one malformed response away from corruption.

Keep dependencies pinned and run an audit in CI. A transitive dependency with a known vulnerability is still a vulnerability in your time-series data aggregation system, and the lockfile is what makes the audit meaningful.

Observability

Emit one structured event per time-series data aggregation operation with the inputs’ shape, the outcome, and a correlation id. Track error rate and latency per component and alert on a step change rather than a single spike.

The correlation id is the thread that ties a user-visible symptom to the time-series data aggregation decision and to the component log entry for the same operation. Without it, an investigation starts from a timestamp and a guess.

Watch the compression encoder failure rate separately from the overall error rate. A jump in compression encoder failures usually means a write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption, not a bug in retention policy, and the two have different owners and different fixes.

Chart the share of time-series data aggregation operations that shed, queue, or succeed. A change in the mix points at the cause faster than an aggregate latency can, because each class has a different root.

Troubleshooting

A write burst that arrives out of order and breaks the monotonically-increasing timestamp assumption. Reproduce with a minimal fixture, then add a guard test so the time-series data aggregation system rejects the condition loudly.

Performance regresses under time-series data aggregation load. Profile with the sampling analyzer, locate the hot path in retention policy, and add a targeted benchmark to lock the regression.

Keep the time-series data aggregation dashboards separate from dependency dashboards. When an operation fails, the correlation id lets you decide in seconds whether the fault is downsample buckets, retention policy, or compression encoder, instead of guessing from a timeout.

Limitations and next steps

• Single-region; multi-region replication is out of scope.
• The downsample buckets is in-process and not horizontally shared by default.
• No auth layer; assume deployment behind an edge that enforces identity.

The time-series data aggregation system is single-region by design; extending to multi-region means making downsample buckets shareable across regions, which is a real change to the consistency model rather than a config flag. The compression encoder is in-process, which is correct for one instance but must move to a shared store before scaling out.

There is no authentication or authorization layer; the system assumes it is deployed behind an edge that enforces identity. Adding one is straightforward but out of scope for this build, and doing it here would distract from the time-series data aggregation correctness that is the point.

The natural next step, once the time-series data aggregation pipeline is trusted in production, is to add a short-TTL cache for idempotent reads and a richer query rollup that exports histograms rather than counters. Both are additive and neither changes the contracts this build establishes.