Trace Sampling
Trace sampling keeps a subset of distributed traces, controlling observability cost while retaining enough data to reason about the system.
Definition
Trace sampling is the set of strategies for deciding which distributed traces to record and retain. Approaches split into head-based — deciding at the root before the outcome is known — and tail-based — deciding after the full trace is assembled. Whatever the method, the sampler must preserve enough of the total population, and record the sampling rate, so aggregate latency and error statistics remain unbiased.
Context and scope
Mandatory in any tracing deployment at scale, because storing every trace of a busy system costs more than the system itself, and a well-chosen sample answers most questions.
Common misconceptions
- Sampling breaks statistics — statistics remain valid if the sampling probability is known and applied consistently; secret or biased sampling is what breaks them
- More sampling is always safer — cost grows linearly while marginal insight falls; the art is choosing where to keep density
- One strategy fits all signals — traces, metrics, and logs each call for a different sampling approach
Anti-patterns
- Sampling without recording the rate, so latency percentiles computed from retained traces are wrong
- Biasing the sample toward a subset of services, hiding whole dependency paths
- Keeping every trace during an incident without any cost control, so the collector itself falls over
Related concepts
Related examples
Authoritative source
What it is
Trace sampling chooses a fraction of traces to store. The decision can be head-based, taken at the root and propagated, or tail-based, taken after the trace completes with its outcome known. Between these poles sit rate-limited, attribute-based, and composite strategies. The shared requirement is that the sampler is consistent and its rate is recorded, so downstream queries can weight the retained traces correctly.
Why it matters
A busy service generates thousands of traces per second; storing all of them is unaffordable and mostly redundant. Sampling makes tracing sustainable, but it must be done carefully: if errors or one service are systematically under-sampled, the resulting metrics are quietly wrong and lead investigations astray. Done right, a small sample still reveals structure, hotspots, and failure paths.
How to choose
Use head sampling as the cheap default, then add tail sampling to keep rare errors and slow traces that the head decision would have discarded. Record the sampling rate on every retained trace, and combine sampling with metrics — metrics carry exact counts, traces carry the details of a few requests. Review the retained rate periodically: as traffic and cost shift, so should the policy.