阅读中文版本

Build a Log Search Service over OpenSearch

A complete log search service: structured ingest, index templates with rollover, time-range queries, aggregations, and retention tests.

log-search/
├── src/ingest.ts
├── src/search.ts
├── src/rollover.ts
├── src/retention.ts
├── tests/search.test.ts
└── package.json

PUT _index_template/logs {
  "index_patterns": ["logs-*"],
  "template": { "settings": { "number_of_shards": 2 }, "mappings": { "properties": { "@timestamp": {"type":"date"}, "level": {"type":"keyword"} } } }
}

await client.bulk({ refresh: false, body: docs.flatMap((d) => [{ index: { _index: 'logs-write' } }, d]) });

{ query: { bool: { filter: [ { range: { '@timestamp': { gte: from, lte: to } } }, { term: { level: 'error' } } ] } } }

"aggs": { "per_minute": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m" } } }

await client.indices.delete({ index: 'logs-' + cutoffDate, ignore_unavailable: true });

Project overview

Logs are cheap to write and expensive to query, and the services that hold them grow without bound unless retention is designed rather than discovered. This build ingests structured logs into daily indices and answers time-bounded queries with a retention policy.

You will define an index template with explicit mappings, bulk-write validated documents into a daily index, query within a time range with filters and aggregations, and roll and delete indices on a schedule.

Time bounding is not an optimization; it is the query contract. A search that can range over all history is a search that will eventually time out, so the endpoint requires a range and the indices make it efficient.

Retention is implemented as index lifecycle rather than as deletes inside one huge index, which keeps deletes cheap and makes the policy a matter of dropping a whole index.

Each phase adds one property: a template, structured ingest, time-range queries, aggregations, and retention, so the service is usable and bounded at every stage.

The service is for operators who have watched a log store grow until queries time out and the disk fills. It makes both problems structural by bounding every query to a range and by expressing retention as an index lifecycle rather than as a delete job.

It is also a compact illustration of why time-series storage and full-text search differ: the same documents that are easy to search are expensive to retain, and the retention policy is as much a part of the design as the query.

What you will build

The service provides five capabilities, each with a test against a fixed document set:

• Index template
• Structured ingest
• Time-range query
• Aggregations
• Retention

A template fixes mappings and an alias; structured ingest validates and bulk-writes into a daily index; time-range queries filter efficiently; aggregations power dashboards; and retention rolls and deletes indices on schedule.

The design keeps ingest off the query-critical path and makes the index lifecycle explicit, so growth is governed by a policy rather than by disk pressure.

Architecture and data model

An ingest endpoint validates and bulk-writes documents into a daily index. A template defines mappings and an index alias with a rollover condition. The search endpoint restricts to a time range, applies filters, and returns aggregations for dashboards.

An ingest endpoint validates each document against a mapping, then bulk-writes it into the daily index behind an alias. Writing through an alias means the query path does not need to know which concrete index holds recent data.

The search endpoint requires a time range and applies filters within it, so a query touches only the indices that overlap the range rather than every index that ever existed.

Aggregations are computed over the same bounded range, which keeps dashboard queries cheap and predictable as history grows.

Retention is expressed as an index lifecycle: roll to a new index on a size or age condition, move older indices to a cheaper tier, and delete them past the retention window.

Writing through an alias decouples the writer from the concrete index name, so rolling to a new index is a lifecycle action rather than a code change and the query path keeps working across the rollover.

The refresh interval is a deliberate trade: shorter makes fresh logs visible sooner and ingest costlier, longer does the reverse, and the right value is a function of the query SLA rather than of a default.

Prerequisites and dependencies

You need a search cluster, a schema definition for the index, and a scheduler for the lifecycle policy.

• Node.js 20 LTS or later
• A reachable OpenSearch or Elasticsearch instance
• Understanding of index mappings

The dependencies cover the HTTP ingest path, the search client, and validation:

• @opensearch-project/opensearch (^2.11.0) — Search and bulk client
• express (^4.21.0) — Ingest and query endpoints

pnpm init
pnpm add @opensearch-project/opensearch express
pnpm add -D typescript tsx vitest

Set the refresh interval to match the query SLA rather than to a default, since a very short interval makes ingest expensive and a very long one makes fresh logs invisible. Record the retention window in configuration and enforce it in the lifecycle policy.

Directory structure

Separate the index template, ingest validation, the query builder, the aggregation definitions, and the lifecycle policy so each is testable independently:

log-search/
├── src/ingest.ts
├── src/search.ts
├── src/rollover.ts
├── src/retention.ts
├── tests/search.test.ts
└── package.json

Phase 1: Index template

Define field mappings and a rollover alias once.

PUT _index_template/logs {
  "index_patterns": ["logs-*"],
  "template": { "settings": { "number_of_shards": 2 }, "mappings": { "properties": { "@timestamp": {"type":"date"}, "level": {"type":"keyword"} } } }
}

Explicit mappings prevent type conflicts and costly reindexing.

Define an index template with explicit field mappings and an alias, so every daily index shares the same shape. Dynamic mapping is convenient and produces the field explosion that makes later aggregations unusable.

Phase 2: Structured ingest

Validate the payload and bulk index into the alias.

await client.bulk({ refresh: false, body: docs.flatMap((d) => [{ index: { _index: 'logs-write' } }, d]) });

Writing to the alias lets rollover move new data transparently.

Validate documents before bulk-writing them, and reject malformed entries without failing the whole batch. Structured ingest is what makes filters and aggregations meaningful rather than best-effort string matching.

Phase 3: Time-range query

Always bound queries by time and add filters.

{ query: { bool: { filter: [ { range: { '@timestamp': { gte: from, lte: to } } }, { term: { level: 'error' } } ] } } }

Unbounded queries scan every shard and are the top cause of outages.

Require a time range on every query and filter within it, so search touches only the relevant indices. A query without a range is the one that eventually times out and takes the cluster with it.

Phase 4: Aggregations

Return per-minute counts or top services for dashboards.

"aggs": { "per_minute": { "date_histogram": { "field": "@timestamp", "fixed_interval": "1m" } } }

Aggregations answer “how many, how often” without shipping all rows.

Expose aggregations over the bounded range to power dashboards, and cap the number of buckets so a high-cardinality field cannot produce an unbounded result.

Phase 5: Retention

Delete indices older than the retention window via the index pattern.

await client.indices.delete({ index: 'logs-' + cutoffDate, ignore_unavailable: true });

Retention caps storage cost and keeps queries fast.

Implement retention as index lifecycle: roll on a condition, move old indices to a cheaper tier, and delete past the window, so growth is governed rather than driven by disk pressure.

Testing

Ingest a fixed set of documents, assert a filtered query returns the right count, that the aggregation buckets match, and that expired indices are selected for deletion.

Ingest a fixed set of documents and assert that a filtered query returns the right count, which proves both the mapping and the query builder agree on the field semantics.

Assert that an aggregation’s buckets match the expected values, since a mapping change can silently change how a field aggregates.

Assert that indices older than the retention window are selected for deletion, which pins the lifecycle policy as behavior rather than as configuration.

Assert that a query without a time range is rejected, because the contract that keeps the cluster healthy should be enforced rather than merely documented.

Assert the mapping by indexing a document with every field type and querying it back, because a mapping surprise usually appears as a subtle type change rather than as an error.

pnpm vitest run

Deployment and operations

Use a dedicated ingest path, refresh interval suited to the query SLA, and separate hot and warm tiers. Never index from the query-critical path synchronously without a queue.

Use a dedicated ingest path and a refresh interval suited to the query SLA, so a burst of logs does not delay the queries that operators rely on during an incident.

Separate hot and warm tiers so recent logs are fast and older logs are cheap, and never index synchronously from a query-critical path without a queue.

Monitor index count, shard count, and disk headroom, because an unmanaged index lifecycle is the most common cause of a cluster turning read-only.

Alert on ingest rejection rate, since silently dropped documents make a search service look complete while missing the events that mattered.

Keep the ingest path separate from the query path and give it its own queue, so a log burst is absorbed rather than passed directly to a cluster that is also serving incident queries.

Alert on the age of the newest document as well as on ingest rate, because a healthy ingest rate with a growing lag means documents are queued but not indexed.

Security considerations

Redact secrets before ingest and restrict read access by tenant. Do not log request bodies containing tokens; scrubbing at the source is cheaper than deleting later.

Treat log content as potentially sensitive and avoid indexing high-cardinality identifiers such as raw user ids when a hash would serve the query.

Restrict ingest with credentials separate from query credentials, so a compromised application cannot rewrite history or delete indices.

Limit aggregation bucket and result sizes, since a crafted high-cardinality query is a resource exhaustion vector against the cluster.

Set retention explicitly rather than keeping logs forever, because a log archive is a growing store of personal data that eventually needs a deletion story.

Restrict deletes to the lifecycle mechanism rather than exposing an index-delete endpoint, because a delete API is a way to erase evidence with a single call.

Redact sensitive fields at ingest where the producer knows them, since the search service cannot reliably identify a secret after the fact.

Observability

Track ingest rate, rejected documents, query latency, and index sizes. Alert on a rising rejection rate, which indicates mapping conflicts.

Track ingest rate, query latency, and the share of queries hitting the time range limit, because the last tells you whether callers are following the contract.

Chart index and shard counts over time, since a steady climb is the earliest signal that the lifecycle policy is not keeping up.

Monitor rejection rate on ingest and the queue depth in front of it, because a growing queue with a stable ingest rate means the cluster cannot keep up.

Measure the age of the newest indexed document, since a growing lag means queries are answering from a stale view even though they succeed.

Track the ratio of query time to the allowed range, because a query that consistently uses only a small fraction of the range suggests the caller is not bounding it as tightly as it could.

Measure shard size distribution, because a few oversized shards are the most common reason a single query is slow while the cluster looks otherwise healthy.

Alert on the ingest queue depth, because a growing queue at a steady ingest rate means the cluster cannot keep up and is about to fall behind.

Troubleshooting

Mapping conflict rejects documents. A field arrived as a string where the template expects a date; normalize at ingest.

Query times out. The query lacked a time range or used wildcard patterns across all indices.

If a query is slow despite a time range, check whether it spans the alias across too many indices, because a wide alias makes a narrow range expensive.

If logs are missing, check the ingest rejection rate before the query, since a rejected document never reaches the index and a successful query cannot find it.

If a field cannot be aggregated as expected, check the mapping first, because a field indexed as text with an analyzed sub-field will not group the way a keyword field does.

Limitations and next steps

• Full-text relevance tuning is not covered.
• No cross-cluster search.
• Retention uses index deletion, not document expiry.

The service ingests structured logs only; parsing arbitrary free-text lines into fields is left to the producer, which keeps ingest predictable but shifts work to clients.

There is no cross-cluster search or long-term archive beyond the retention window, so a compliance requirement for years of logs would need a separate cold store.

Aggregations are over the same index set as the query and have no materialized view, so a very expensive dashboard aggregation is recomputed on every render.

The lifecycle policy is time and size based and does not account for query value, so a rarely queried index is retained exactly as long as a busy one.

There is no anomaly detection or alerting on the log content itself; the service stores and searches, and alerting is left to the operator’s own rules.

Aggregations have no materialized view, so a dashboard that recomputes a wide aggregation on every refresh will scale poorly with the number of viewers.