Adaptive Sampling Strategies
Adaptive sampling varies the sampling rate by the value of the data — sampling errors and slow spans heavily, healthy traffic lightly — so the retained set is informative within a fixed budget.
Definition
Adaptive sampling is the technique of varying the probability of retaining a telemetry record based on its attributes — high rate for errors and slow requests, low rate for fast healthy ones — so that within a fixed storage budget the retained sample contains disproportionately the records that explain incidents.
Context and scope
Applied in distributed tracing (tail-based sampling), log retention, and any telemetry pipeline where the interesting records are rare and the budget is fixed.
Common misconceptions
- Adaptive sampling is just probabilistic sampling — probabilistic sampling uses one rate; adaptive uses a rate per stratum, preserving the rare interesting strata
- Adaptive sampling biases the data — it biases retention toward the interesting; the analysis must weight by inverse sampling probability to remain unbiased
Anti-patterns
- Sampling errors at 100% and healthy at 0%, so the error rate cannot be computed because the denominator is unsampled
- Changing the sampling rate without recording it on each span, making the retained set uninterpretable
Related concepts
Authoritative source
What it is
Adaptive sampling is the technique of varying the probability of retaining a telemetry record based on its attributes — high rate for errors and slow requests, low rate for fast healthy ones — so that within a fixed storage budget the retained sample contains disproportionately the records that explain incidents. Applied in distributed tracing (tail-based sampling), log retention, and any telemetry pipeline where the interesting records are rare and the budget is fixed. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.