Trace Sampling Policy
A trace sampling policy decides which traces to keep within a budget — head sampling by probability, tail sampling by attributes — so the retained set is informative and affordable.
Definition
A trace sampling policy is the configuration of which traces are retained: head sampling makes the decision at the root span by probability, tail sampling makes it at the end by attributes (errors, slow duration, specific endpoints), so that the retained set is biased toward the informative traces within a fixed storage budget.
Context and scope
Applied in distributed tracing, OpenTelemetry collectors, and any tracing pipeline where the volume exceeds the budget and the interesting traces are rare.
Common misconceptions
- Head sampling is enough — it is enough for volume; it samples uniformly, so rare errors are sampled at the same rate as healthy traffic and often missed
- Tail sampling has no overhead — it buffers the trace to evaluate, which adds memory and latency at the collector; the overhead is the cost of intelligence
Anti-patterns
- Head sampling at 1% and missing every error trace because errors are 0.1% of traffic
- Tail sampling with so many rules that the evaluation cost exceeds the collection cost
Related concepts
Authoritative source
What it is
A trace sampling policy is the configuration of which traces are retained: head sampling makes the decision at the root span by probability, tail sampling makes it at the end by attributes (errors, slow duration, specific endpoints), so that the retained set is biased toward the informative traces within a fixed storage budget. Applied in distributed tracing, OpenTelemetry collectors, and any tracing pipeline where the volume exceeds the budget and the interesting traces are rare. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.