Log Sampling
Log sampling keeps a representative subset of log events, preserving patterns and errors while dropping the redundant volume that inflates cost.
Definition
Log sampling is the practice of retaining only a fraction of log events — often keyed by a stable signature such as the message template or the emitting service — so that the statistical shape of the logs is preserved while the stored volume is bounded. It differs from log dropping: a sampled stream remains analysable because the sampling probability is known, but a discarded stream is gone. Errors and rare events are usually exempted or always kept.
Context and scope
High-throughput services that emit millions of similar lines per hour, where per-line cost dominates the observability budget but the repeated lines carry little marginal information.
Common misconceptions
- Sampling loses the incident — if sampling is signature-aware and errors are pinned, the incident is preserved; naive uniform drops are what lose it
- Fewer logs is always cheaper overall — missed detail can cost more in downtime than the storage saved
- Sampling and aggregation are the same — aggregation keeps counts; sampling keeps individuals, which matters for message content
Anti-patterns
- Sampling uniformly at random so a rare but critical error disappears
- Enabling sampling in production without recording the sampling rate, making counts unusable
- Dropping logs at the source with no local buffering, so the last seconds before a crash vanish
Related concepts
Related examples
Authoritative source
What it is
Log sampling selects events to keep by a rule rather than keeping all or none. A common approach keys on a message signature — the template with variable parts removed — and keeps the first occurrence plus a fraction of repeats, so a new message is always visible while a storm of identical lines is thinned. The sampler records its rate so downstream systems can extrapolate counts.
Why it matters
Logs are the signal most likely to grow without bound, because developers add lines freely and every line is stored. Much of that volume is repetition: the same retry, the same heartbeat. Sampling collapses the repetition while preserving the fact that it happened and roughly how often, cutting storage and index cost without blinding the operator to rare events.
How to use it well
Pin errors and rare signatures to always keep, sample only the high-frequency repeats, and always attach the sampling rate to each event so counts can be reconstructed. Prefer sampling on structured fields over free text, and keep enough local buffer to capture the seconds before a crash. Measure what sampling removed so you can prove, not assume, that no important class was lost.