阅读中文版本

Exemplars Linking Metrics to Traces

Exemplars attach sampled trace identifiers to metric points, letting an operator jump from a latency spike directly to a representative request trace.

Definition

An exemplar is a sampled trace identifier or other metadata attached to a metric observation, so an aggregated timeseries can be linked back to concrete requests.

It is used to shorten the path from a dashboard alert to a root cause trace without searching logs blindly. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.

When to use it

A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.

How it works

Metrics aggregate and lose identity, so an exemplar preserves a few concrete examples, typically a sampled trace identifier, alongside each bucket.

The value depends on sampling correlated with the tracing pipeline, so both systems must agree on the trace identifier and configure sampling that captures interesting requests.

Trade-offs and alternatives

Exemplars add cardinality and storage cost, so they are enabled selectively on metrics that matter for diagnosis. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.

Common misconceptions

A persistent misconception is that exemplars record every request rather than a sample. Another is that they work without a tracing backend. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.

Anti-patterns to avoid

Two recurring anti-patterns are to attaching exemplars only to success paths and missing the slow ones and to sampling so sparsely that no exemplar represents the spike. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Operational Dashboard Design, Selecting Service Level Indicators, On-Call Rotation Design.