Metrics Exemplars
Exemplars attach representative trace ids to histogram buckets, linking the aggregate metric to concrete slow requests — from p99 spike to an actual trace.
Definition
An exemplar is a sample measurement — value, timestamp, and trace or span id — attached to a histogram bucket or summary quantile. When a request lands in the slow bucket, its trace id rides along, letting you click from the metric spike to an exemplifying trace. Exemplar selection keeps a bounded sample per bucket, preferring recent or extreme observations.
Context and scope
Supported by Prometheus exemplar storage and OpenTelemetry exemplar APIs. Applied on latency and size histograms where aggregate spikes need instance-level diagnosis.
Common misconceptions
- Exemplars store all samples — they keep a bounded, selected subset per bucket; the aggregate stays exact, samples are representative
- They replace trace sampling — they complement it: exemplars point into whatever traces sampling retained
- Any metric can carry exemplars — they attach to histogram buckets and quantiles, where asking which request produced this value is a meaningful question
Anti-patterns
- Storing exemplars with high-cardinality span ids in systems without exemplar support and exploding storage
- Clicking exemplar links into traces the sampling policy already dropped
- Enabling exemplars on every histogram and multiplying scrape payloads for rarely used metrics
Related concepts
Related examples
Authoritative source
What it is
A histogram says the p99 bucket grew. A trace says one request took 900 milliseconds. Exemplars are the join: bucket entries carry trace ids, so the aggregate and the instance reference each other.
How it works
During measurement, the instrumentation optionally records exemplars — value, time, trace and span ids — into bucket metadata. Storage keeps them bounded per bucket. Visualization surfaces exemplar links on dashboards; clicking opens the referenced trace in the tracing UI, subject to that trace having been sampled.
When to enable
Latency histograms on user-facing endpoints, queue lag metrics, and any aggregate whose spikes trigger “find me one instance” investigations. The payoff is diagnosis speed: from alert to representative trace in one click.
Pitfalls
Exemplar links rot when retention differs: metric retention of weeks versus trace retention of days leaves dead links. Tail sampling must keep interesting traces, or exemplars point at nothing. Per-bucket exemplar storage adds scrape size — enable where diagnosis value concentrates.