阅读中文版本

API p99 Latency Reduction Workflow

Reduce tail latency by finding the slowest requests and fixing the causes of outliers rather than averages.

Overview

Reduce p99 API latency by targeting the causes of tail outliers.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• Latency histograms or tracing
• Distributed traces
• A load test

Procedure

1. Locate the tail

Find the time windows and endpoints where p99 latency spikes.

Expected state: Tail latency hotspots are identified.

2. Inspect slow traces

Examine traces of the slowest requests to find where time is spent.

Expected state: The slow span or dependency is identified.

3. Fix the outlier cause

Address queueing, lock contention, GC, or slow dependencies causing the tail.

Expected state: The specific outlier cause is mitigated.

4. Verify the tail improves

Re-measure p99 under load and confirm it dropped without hurting p50.

Expected state: The tail improved without a median regression.

# find endpoints with a long p99
histogram_query 'api_latency' --quantile 0.99 --group-by endpoint

Verification

Confirm p99 drops under load while p50 stays stable or improves.

Decision points

Optimise average or tail?

• Target the tail
• Optimise the average

Recommendation: Target the tail; averages hide the experience of the slowest users.

Sampling or full traces on tail?

• Tail-based sampling
• Uniform sampling

Recommendation: Use tail-based sampling to keep the slow requests you need.

Common failures

• Averages hide tail problems.
• Sampling misses the rare slow requests.
• Fixing p50 does not improve worst-case experience.