Latency Percentiles and Histograms
Latency percentiles summarize distributions from histograms — p50, p99, p999 — and correct aggregation, bucket boundaries, and coordinated-omission correction decide whether they are trustworthy.
Definition
Percentile reporting turns raw latency samples into distribution summaries. Correct practice: record into high-dynamic-range histograms, aggregate by merging histograms rather than averaging percentiles, choose bucket boundaries matching SLO thresholds, and correct for coordinated omission where the measurement loop pauses during slow responses and under-samples the tail.
Context and scope
Implemented with HdrHistogram and OpenTelemetry histogram metrics. Feeds SLO burn-rate alerts and performance regression gates in CI.
Common misconceptions
- Averaging per-instance percentiles yields the fleet percentile — percentile aggregation is not associative; merge histograms or use t-digest sketches instead
- More buckets always mean better accuracy — value lies in bucket boundaries aligned to thresholds; arbitrary fine buckets misallocate resolution
- p99 from one hour describes every minute — bursts hide inside hourly aggregates; window size must match the user experience timescale
Anti-patterns
- Computing a global p99 by averaging node-level p99s and underreporting the tail
- Measuring latency from a request loop that waits for responses, missing exactly the slow ones (coordinated omission)
- Alerting on p50 and treating percentile spikes that do not breach the mean as invisible
Related concepts
Related examples
Authoritative source
What it is
A latency dataset is a distribution. Percentiles communicate the distribution to humans and SLOs; histograms make percentiles computable across time and space without storing every sample.
How it works
Samples land in log-encoded buckets spanning microseconds to minutes at constant relative error. Percentiles read off cumulative counts; aggregation merges bucket counts, preserving correctness across nodes. Coordinated omission correction compensates for the requests the measurement loop failed to issue while blocked on slow responses, restoring the tail the loop itself hid.
When to apply
Any service latency SLO, CI performance gates, and capacity dashboards. Export histogram metrics rather than pre-computed percentiles so downstream aggregation stays valid.
Pitfalls
Percentiles are not additive or averagable — the fleet p99 comes from merged histograms. Bucket boundaries must align with thresholds, or a p99 read interpolates across an over-wide bucket. Very high percentiles on low traffic are sampling artifacts: require minimum counts.