Telemetry Sampling
Sampling keeps a representative subset of traces, logs, or metrics instead of every event, cutting cost and volume while preserving statistically useful visibility.
Definition
Telemetry sampling is the deliberate selection of a subset of events from a stream by probability, rate, attribute, or tail criteria, so observability cost and volume stay bounded while key signals remain measurable.
Context and scope
Used when telemetry volume outgrows budget or backend capacity, especially in high-throughput services and distributed traces.
Common misconceptions
- Sampling is the same as aggregation; a sample still needs weighting to estimate totals
- A one percent sample reports exact error counts without scaling
- Sampling is only for traces; logs and metrics also need volume policies
Anti-patterns
- Dropping data randomly with no recorded sampling rate
- Sampling away all errors together with routine traffic
- Building alerts on sampled spans instead of complete counters
Related concepts
Related examples
Authoritative source
How it works
Head-based sampling decides at the start of a request, usually by hashing the trace ID against a rate, so every service makes the same decision and traces stay complete. Tail-based sampling buffers a trace at a collector and decides after seeing it, which lets you keep all errors and slow requests while dropping routine traffic. Metrics are usually aggregated with weights rather than sampled.
When to use it
Sampling is appropriate when telemetry volume exceeds budget or backend capacity, especially for high-throughput traces and debug logs. It is not appropriate for billing, audit, or security records that must be complete. Compared with dropping data blindly, a documented sampling policy preserves known-important events and records the rate.
Trade-offs
Rare events can disappear, so error and outlier policies matter more than the headline rate. Dashboards built on sampled traces need scaling weights or they undercount. Tail sampling requires buffering and enough collector capacity; head sampling is cheap but cannot know the outcome.
Example
A checkout service samples one percent of traces at the SDK and keeps every trace containing an error at the collector. When latency spikes, engineers still see failing requests in full, and trace IDs connect those requests to logs that were also sampled consistently.
Misconceptions
Sampling is not aggregation; a one percent sample cannot report exact error totals without weighting. Keeping only errors is not enough, because you also need a baseline of successful traffic. And sampling does not excuse missing alerts: detectors should run on complete counters, not sampled spans.
Related concepts
A telemetry pipeline applies sampling centrally; distributed tracing depends on consistent decisions; log levels determine which records are candidates for sampling.