Tail Sampling
Tail sampling decides whether to keep a trace after collecting all its spans, so complete slow or failed traces are retained and routine ones dropped.
Definition
Tail sampling is a distributed-tracing sampling strategy that buffers all spans of a trace until the trace is judged complete, then decides whether to keep it based on the whole — its latency, error status, or other attributes. Because the decision sees the full trace, it can keep every slow or failed request while sampling only a fraction of fast successes, unlike head sampling, which must guess before the outcome is known.
Context and scope
Used in tracing pipelines where the interesting traces are rare — tail latency and errors — and a uniform sample would miss most of them.
Common misconceptions
- Tail sampling is strictly better than head sampling — it needs buffering and a collector that sees all spans, adding cost and complexity
- Just raise the tail-sampling rate — keeping more slows the backend and the bill; the point is to be selective
- Any collector can do it — tail sampling requires routing all spans of a trace to the same decision point
Anti-patterns
- Dropping a span before the decision, so the buffered trace is incomplete and misjudged
- Tuning policies to keep everything, which defeats the purpose and overloads the store
- Collecting all spans at full fidelity in every region when a central decision point is needed
Related concepts
Related examples
Authoritative source
What it is
Head sampling decides at trace start whether to record, before anything is known about the outcome. Tail sampling waits: all spans of a trace are buffered for a bounded window, and a policy evaluates the assembled trace — duration, status codes, attributes — to keep or discard it. The result is a sample biased toward exactly the traces an engineer would open by hand.
Why it matters
Failures and tail-latency outliers are rare relative to the steady stream of fast successful requests. A uniform one-percent sample rarely contains the very incident under investigation. Tail sampling inverts the ratio: keep one hundred percent of errors and slow traces, and a small fraction of the rest, so the retained data is dense where it matters and cheap where it does not.
How to use it well
Buffer spans long enough to cover the slowest traces, but not so long that memory grows unbounded, and route all spans of a trace to one decision point — a central collector or sharding by trace id. Combine policies: keep errors and high latency, sample successes at a rate, and always keep traces that carry a specific attribute. Emit a sampled-rate counter so the backend can reweight its statistics correctly.