Head Versus Tail Sampling
Tail sampling decides trace retention after the trace completes — keeping all errors and slow traces while dropping a fraction of healthy ones — trading streaming cost for representative diagnostics.
Definition
Head sampling decides at trace start whether to keep it, independent of outcome. Tail sampling collects spans for every trace, evaluates the complete trace at completion — duration, error status, attribute filters — and retains according to policy: all errors, all slow traces, a percentage of the rest. The cost is buffering and routing the full stream through a collector tier.
Context and scope
Implemented in OpenTelemetry collectors and tracing backends with tail sampling processors. Required when error traces matter and head sampling at low rates randomly misses them.
Common misconceptions
- Sampling is to save storage — it saves ingestion cost and privacy exposure; storage is the smaller half
- Tail sampling requires dropping nothing at the source — spans must all reach the sampling tier; spans lost before cannot be recovered by any policy
- A low head rate approximates tail sampling — random low-rate head sampling drops errors and latency outliers at the same rate as everything else — exactly the traces you keep tails for
Anti-patterns
- Enabling tail sampling behind load balancers that break trace assembly and getting partial-trace decisions
- Keeping zero percent of healthy traces and losing the baseline that makes anomalies interpretable
- Buffering entire trace streams without backpressure and OOMing the collector tier during bursts
Related concepts
Related examples
Authoritative source
What it is
You cannot keep every trace, but you cannot know at trace start whether a trace will be interesting. Head sampling guesses early and cheaply; tail sampling decides late with the full picture.
How it works
The collector tier holds spans grouped by trace id until the trace completes or a timeout expires. Policy evaluates the whole: duration thresholds, error flags, attribute matches. Retention rates are per policy — 100 percent of errors, 100 percent over 1 second, 5 percent otherwise. Order of policy evaluation matters for combined decisions.
When to choose
Whenever error and outlier traces must exist in the sample: production diagnosis, SLI validation against traces, exemplar link targets. Head sampling suits cost floors where completeness matters less.
Pitfalls
The collector becomes the scaling unit — it processes the unsampled stream, sized like the production traffic it observes. Trace assembly requires consistent span routing: load balancers hashing by connection rather than trace id split traces and corrupt decisions. Long-running traces outlive buffers — deadline flush policies decide their fate.