Centralised Log Aggregation
Centralised log aggregation ships logs from every instance into one searchable store, replacing per-host files with queryable, correlated records.
Definition
Log aggregation is the collection, shipping, and indexing of log records from many sources into a central system so they can be searched, correlated, and retained beyond a single host.
Context and scope
It is essential once a service runs on more than one instance, because a request may touch hosts the operator cannot predict.
Common misconceptions
- logging everything verbatim improves debuggability
- centralised logs can store every field without cost concerns
Anti-patterns
- shipping raw request payloads including personal data
- raising a volume that makes the index expensive and slow to query
Related concepts
Related examples
Authoritative source
Definition
Log aggregation is the collection, shipping, and indexing of log records from many sources into a central system so they can be searched, correlated, and retained beyond a single host.
It is essential once a service runs on more than one instance, because a request may touch hosts the operator cannot predict. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.
When to use it
A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.
How it works
Structured fields such as request identifiers, tenant, and latency make logs queryable, whereas free-form messages force fragile text searching.
Volume drives both cost and query latency, so sample verbose records and drop noise rather than indexing everything by default.
Trade-offs and alternatives
Logs must respect privacy constraints: redact secrets and personal data at the source, before shipping, because a central store widens access. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.
Common misconceptions
A persistent misconception is that logging everything verbatim improves debuggability. Another is that centralised logs can store every field without cost concerns. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.
Anti-patterns to avoid
Two recurring anti-patterns are to shipping raw request payloads including personal data and to raising a volume that makes the index expensive and slow to query. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include The USE Method for Resource Analysis, Symptom-Based Alerting, Trace Context Propagation.