Telemetry Cardinality Control
Telemetry cardinality control bounds the number of distinct label combinations, so a high-cardinality dimension does not explode storage and make the metric unreadable as one line per value.
Definition
Telemetry cardinality control is the practice of limiting the distinct label value combinations on a metric or log, so that a dimension like user_id or request_id — which can take millions of values — does not create a separate time series per value, exploding storage and making dashboards unusable.
Context and scope
Applied in Prometheus, statsd, and any metrics system where the cost of a time series is non-trivial and an unbounded label can multiply the series count by the cardinality.
Common misconceptions
- High cardinality is always bad — it is bad for metrics (one series per value); it is fine for logs and traces, which are designed for it
- Dropping labels fixes cardinality — it fixes the count but loses the dimension; the fix is to bucket or to move the high-cardinality data to logs
Anti-patterns
- Putting user_id as a metric label, creating a time series per user and exhausting memory in hours
- Bucketing a high-cardinality label into so few buckets that the dimension is useless for diagnosis
Related concepts
Authoritative source
What it is
Telemetry cardinality control is the practice of limiting the distinct label value combinations on a metric or log, so that a dimension like user_id or request_id — which can take millions of values — does not create a separate time series per value, exploding storage and making dashboards unusable. Applied in Prometheus, statsd, and any metrics system where the cost of a time series is non-trivial and an unbounded label can multiply the series count by the cardinality. The goal is not to collect all data but to be able to answer the question you did not think to ask in advance — which requires high-cardinality labels and exemplars that link traces to the metric that triggered them.
How it works
Three signals compose: metrics aggregate over time, traces preserve causality across boundaries, and logs carry the unstructured detail. Sampling is adaptive — head sampling for known-interesting paths, tail sampling to retain slow or error traces at a fixed budget. Cardinality is controlled at the source, not the query, because a high-cardinality label explodes storage and makes dashboards unusable.
When to invest and pitfalls
Invest when mean time to detect matters — in production, at scale, with paying users. The pitfall is instrumenting everything and alerting on nothing: dashboards full of lines that no one reads, alerts that fire on noise and are then muted, leaving the real outage undetected. Symptom-based alerts on user-visible degradation outperform cause-based alerts on internal thresholds.