阅读中文版本

Build an Inverted Index Search Engine

Inverted Index Search Engine — a complete walkthrough of full-text search, covering tokenizer pipeline, posting list writer, tf-idf ranker, and testing.

inverted_index_search/
├── src/index.ts
├── src/tokenizer-pipeline.ts
├── src/posting-list-writer.ts
├── tests/inverted_index_search.test.ts
└── package.json

// Tokenizer pipeline: full-text search core
export function tokenizerpipeline(input: TokenizerpipelineInput): TokenizerpipelineResult {
  // Build the tokenizer pipeline that anchors full-text search and defines the core data shape.
  return processTokenizerpipeline(input);
}

// Posting list writer — handles full-text search Add the posting list writer
export class Postinglistwriter {
  constructor(private opts: PostinglistwriterOptions) {}
  async run(ctx: Context): Promise<PostinglistwriterOutput> {
    return this.execute(ctx);
  }
}

// TF-IDF ranker: full-text search stage
export const tfidfranker = (next: Next) => async (req: Request) => {
  const out = await transform(req);
  return next(out);
};

// Merge segment for full-text search
export async function mergesegment(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
  let acc = init();
  for await (const c of stream) acc = merge(acc, c);
  return acc;
}

// Query parser: full-text search finalization
export function queryparser(state: State): Result {
  validate(state);
  return finalize(state);
}

Project overview

This project builds a full-text search system whose reason for existing is resolving boolean and ranked queries against a corpus that is indexed incrementally as documents arrive. The temptation in full-text search is to bolt the interesting part onto a working prototype and discover the hard cases in production; this build inverts that order and treats a rare term whose posting list is a single long array that dominates query latency for phrase queries as the first thing designed, not the last.

You will implement Tokenizer pipeline, Posting list writer, TF-IDF ranker, Merge segment, and Query parser, each as a module with one responsibility and a test that proves it in isolation. The full-text search pipeline is then the wiring that composes them, and the integration test is where the composition is shown to survive a rare term whose posting list is a single long array that dominates query latency for phrase queries.

The walkthrough is written for a developer who has used a full-text search system but has not had to make one honest. It favors explicit data shapes over clever abstractions, and it names the failure each design choice prevents rather than presenting the choice as a style preference.

Each phase ends with something observable — a tokenizer pipeline that returns the right shape, a posting list writer that rejects bad input, an integration test that goes green — so you can stop at any point and still have a full-text search system that works for the cases covered so far.

What you will build

The finished full-text search system exposes five capabilities, each with a focused test:

• Tokenizer pipeline
• Posting list writer
• TF-IDF ranker
• Merge segment
• Query parser

Tokenizer pipeline is the foundation that every later stage assumes; Posting list writer is where full-text search correctness lives; TF-IDF ranker handles a rare term whose posting list is a single long array that dominates query latency for phrase queries; Merge segment makes the system observable; and Query parser is the graceful-degradation layer that keeps overload from becoming corruption.

The build keeps the full-text search logic in ordinary modules you can read in one sitting, and pulls in dependencies only for the parts that are genuinely tedious to hand-roll. That keeps the diff between the tutorial and a real codebase small, and keeps the failure modes visible in code rather than hidden behind a framework.

Architecture and data model

The system is organized around Tokenizer pipeline, Posting list writer, and TF-IDF ranker, with Merge segment tying them together. Data flows as an inverted index of term-to-posting-list mappings, segmented by document id range and merged in the background.

Each component is a module with a pure interface where practical: the same input produces the same output, and side effects are pushed to the edges. That discipline is what makes the phases independently testable, because a test for TF-IDF ranker does not need to stand up Tokenizer pipeline and Posting list writer to exercise the interesting branch.

The data model is deliberately explicit. An inverted index of term-to-posting-list mappings, segmented by document id range and merged in the background Keeping the shape written down — in a schema, a type, or a fixture — means a change to the shape is a visible diff rather than a silent runtime shift, and it means the integration test can assert against a concrete shape instead of guessing.

Configuration and code are kept apart: the configuration says what full-text search should do, the modules say how. A reviewer can change a threshold or a policy without reading the implementation, which keeps the risky changes small and lets the diff stand as the review.

The failure path through TF-IDF ranker is designed before the happy path is finished, because a rare term whose posting list is a single long array that dominates query latency for phrase queries is the case that distinguishes a full-text search system from a demo. Every component declares what it does when its inputs are wrong, and the integration test asserts each of those declarations.

Prerequisites and dependencies

Have these ready before you install anything, so the full-text search setup is not blocked by a missing tool:

• Node.js 20 LTS or later
• Familiarity with full-text search fundamentals
• A local environment to run the test suite

The runtime dependencies are chosen to keep the full-text search datapath legible:

• express (^4.21.0) — Search API
• minisearch (^7.0.0) — In-memory index baseline
• stemmer (^1.0.0) — Token stemming

# setup for inverted-index-search
pnpm init
pnpm add express minisearch stemmer
pnpm add -D typescript tsx vitest

Versions are pinned to a major range so the full-text search install is reproducible; commit the lockfile. The tokenizer pipeline reads its operational parameters from the environment and validates them at startup, so a missing value fails the boot rather than the first full-text search request.

Directory structure

Keep each full-text search component in its own file so it can be tested without starting the others:

inverted_index_search/
├── src/index.ts
├── src/tokenizer-pipeline.ts
├── src/posting-list-writer.ts
├── tests/inverted_index_search.test.ts
└── package.json

Phase 1: Tokenizer pipeline

Build the tokenizer pipeline that anchors full-text search and defines the core data shape.

// Tokenizer pipeline: full-text search core
export function tokenizerpipeline(input: TokenizerpipelineInput): TokenizerpipelineResult {
  // Build the tokenizer pipeline that anchors full-text search and defines the core data shape.
  return processTokenizerpipeline(input);
}

After this phase the tokenizer pipeline works in isolation and is covered by a unit test. The tokenizer pipeline defines the data shape that every later stage assumes, so getting it right first means the later phases compile against a stable contract.

Treat the tokenizer pipeline as the single source of truth for the full-text search data shape. Validate inputs at the boundary and reject early, so a malformed value never reaches Posting list writer or TF-IDF ranker where its effect would be harder to localize.

Phase 2: Posting list writer

Add the posting list writer so full-text search can resolving boolean and ranked correctly.

// Posting list writer — handles full-text search Add the posting list writer
export class Postinglistwriter {
  constructor(private opts: PostinglistwriterOptions) {}
  async run(ctx: Context): Promise<PostinglistwriterOutput> {
    return this.execute(ctx);
  }
}

After this phase the posting list writer works in isolation and is covered by a unit test. This is where full-text search correctness lives: the posting list writer is what makes resolving boolean and ranked queries against a corpus that is indexed incrementally as documents arrive true, and its test is the proof.

Keep the posting list writer pure where you can. A function that takes an input and returns an output is trivial to test and impossible to mock wrong; a class with hidden state is the opposite. Push side effects to the caller and keep the posting list writer about the transformation.

Phase 3: TF-IDF ranker

Wire the tf-idf ranker to handle a rare term whose without losing data.

// TF-IDF ranker: full-text search stage
export const tfidfranker = (next: Next) => async (req: Request) => {
  const out = await transform(req);
  return next(out);
};

After this phase the tf-idf ranker works in isolation and is covered by a unit test. The tf-idf ranker is the difference between a full-text search system that works on the happy path and one that survives a rare term whose posting list is a single long array that dominates query latency for phrase queries.

Design the failure before the success. Decide what tf-idf ranker does when a rare term whose posting list is a single long array that dominates query latency for phrase queries: reject, retry, shed, or degrade — and write that test first. A full-text search system whose failure behavior is unspecified will pick one at random under pressure, and it is usually the wrong one.

Phase 4: Merge segment

Integrate the merge segment to make full-text search observable and testable end to end.

// Merge segment for full-text search
export async function mergesegment(stream: AsyncIterable<Chunk>): Promise<Aggregate> {
  let acc = init();
  for await (const c of stream) acc = merge(acc, c);
  return acc;
}

After this phase the merge segment works in isolation and is covered by a unit test. Observability is not a finishing touch; it is how you know the full-text search system is doing what it claims in production.

Emit one structured event per meaningful action, with enough context to localize a failure without a second round-trip. The merge segment is what turns “something is slow” into “the posting list writer is slow for inputs matching this shape,” which is the difference between an alert and an investigation.

Phase 5: Query parser

Finalize the query parser so the full-text search system degrades gracefully under overload.

// Query parser: full-text search finalization
export function queryparser(state: State): Result {
  validate(state);
  return finalize(state);
}

After this phase the query parser works in isolation and is covered by a unit test. The query parser is what keeps overload from becoming corruption: when the full-text search system is asked for more than it can do, it degrades in a defined way instead of failing in an undefined one.

Pick a degradation strategy for query parser — shed, queue, or backpressure — and make it explicit in code and in a test. Leaving the behavior to chance is how a full-text search traffic spike becomes a data loss incident.

Testing

Each phase has a focused test: tokenizer pipeline, posting list writer, and tf-idf ranker are exercised in isolation, then an integration test drives the full full-text search pipeline and asserts the a rare term whose case is handled.

Start with unit tests that pin each component’s contract: the tokenizer pipeline returns the documented shape, the posting list writer transforms correctly, and the tf-idf ranker fails in the documented way. These tests are fast and they localize regressions to the component that broke.

Then add an integration test that composes the components and drives a realistic full-text search scenario end to end. Assert the happy path first, then deliberately trigger a rare term whose posting list is a single long array that dominates query latency for phrase queries and assert the system degrades rather than corrupts. Because the components are pure where practical, the integration test stays deterministic rather than flaky.

A full-text search suite must also prove the negative: bad input is rejected at the boundary, overload is shed rather than queued indefinitely, and a partial failure does not leave the system in an inconsistent state. These are the tests that catch the regressions a happy-path suite never would.

The entire full-text search suite runs with no external service; every dependency is either in-process or stubbed, so every case is reproducible in CI. That is what lets the tf-idf ranker failure-mode assertions live as tests rather than as a manual checklist that drifts out of date.

# run the full-text search test suite
pnpm vitest run

Deployment and operations

Run at least two full-text search instances behind a load balancer and share state in Redis. Set resource limits so full-text search overload surfaces as backpressure rather than failure.

Run at least two stateless instances behind a load balancer, and move any shared state into a shared store so the full-text search behavior is global rather than per instance. Size limits so the system sheds load before it runs out of memory, because a full-text search system that OOMs under load fails in a way that is hard to reason about.

Drain on deploy: stop accepting new work, let in-flight full-text search operations finish, then exit. Because the system holds no state that cannot be reconstructed, a rollout is a matter of cycling instances one at a time while watching the error rate.

When a dependency is unavailable, decide explicitly whether the full-text search system fails closed or degrades. Leaving that choice to chance is how a dependency outage becomes a silent data loss, so make the choice visible in code and in a test.

Security considerations

Validate all full-text search inputs at the boundary, reject oversized payloads before buffering them, and keep secrets in the environment rather than the repository.

The full-text search system is a choke point, so it is the right place to reject malformed input before it reaches posting list writer or tf-idf ranker. Validate the shape and the size at the edge, and let the inner components assume what they receive is well-formed.

Never log secrets or full payloads; log enough to localize a failure without recording data that should not be on disk. Treat any external call as untrusted and validate its response before using it, because a full-text search system that trusts its inputs blindly is one malformed response away from corruption.

Keep dependencies pinned and run an audit in CI. A transitive dependency with a known vulnerability is still a vulnerability in your full-text search system, and the lockfile is what makes the audit meaningful.

Observability

Emit one structured event per full-text search operation with the inputs’ shape, the outcome, and a correlation id. Track error rate and latency per component and alert on a step change rather than a single spike.

The correlation id is the thread that ties a user-visible symptom to the full-text search decision and to the component log entry for the same operation. Without it, an investigation starts from a timestamp and a guess.

Watch the tf-idf ranker failure rate separately from the overall error rate. A jump in tf-idf ranker failures usually means a rare term whose posting list is a single long array that dominates query latency for phrase queries, not a bug in posting list writer, and the two have different owners and different fixes.

Chart the share of full-text search operations that shed, queue, or succeed. A change in the mix points at the cause faster than an aggregate latency can, because each class has a different root.

Troubleshooting

A rare term whose posting list is a single long array that dominates query latency for phrase queries. Reproduce with a minimal fixture, then add a guard test so the full-text search system rejects the condition loudly.

Performance regresses under full-text search load. Profile with the sampling analyzer, locate the hot path in posting list writer, and add a targeted benchmark to lock the regression.

Keep the full-text search dashboards separate from dependency dashboards. When an operation fails, the correlation id lets you decide in seconds whether the fault is tokenizer pipeline, posting list writer, or tf-idf ranker, instead of guessing from a timeout.

Limitations and next steps

• Single-region; multi-region replication is out of scope.
• The tokenizer pipeline is in-process and not horizontally shared by default.
• No auth layer; assume deployment behind an edge that enforces identity.

The full-text search system is single-region by design; extending to multi-region means making tokenizer pipeline shareable across regions, which is a real change to the consistency model rather than a config flag. The tf-idf ranker is in-process, which is correct for one instance but must move to a shared store before scaling out.

There is no authentication or authorization layer; the system assumes it is deployed behind an edge that enforces identity. Adding one is straightforward but out of scope for this build, and doing it here would distract from the full-text search correctness that is the point.

The natural next step, once the full-text search pipeline is trusted in production, is to add a short-TTL cache for idempotent reads and a richer merge segment that exports histograms rather than counters. Both are additive and neither changes the contracts this build establishes.