Triage a Performance Regression
Confirm a latency or throughput regression against a baseline, narrow it to a change or dependency, mitigate safely, and add a guardrail that catches the next one.
Objective
Move from a vague "the app feels slow" report to a measured regression, an identified cause, a safe mitigation, and a permanent detection signal.
Environment
- A load-testing environment or a canary with production-like traffic
- Baseline latency and throughput metrics with percentile breakdowns
- Deploy history and dependency change logs
- Profiling or tracing available for the affected service
Steps
1. Define the regression precisely
Identify the affected route, percentile, time window, and comparison baseline. Separate latency, throughput, error rate, and saturation instead of calling everything "slow".
Expected state: The regression has a metric, a magnitude, and a start time.
2. Reproduce under controlled load
Run the same workload against the current and previous builds in a non-production environment, or compare canary and stable cohorts in production.
Expected state: The difference reproduces with the same load profile, or the production cohort comparison shows a stable gap.
3. Narrow the change window
Correlate the start time with deploys, feature flags, dependency upgrades, traffic mix changes, and database migrations. Use git bisect or canary cohorts when the window contains several changes.
Expected state: One change or dependency is the leading hypothesis, with evidence.
4. Mitigate before root cause
Roll back, disable the flag, shed the expensive path, or scale the bottleneck, depending on risk. Preserve the evidence and avoid masking the regression with a larger instance without understanding it.
Expected state: User-facing latency returns to the target while investigation continues.
5. Fix and verify
Implement the fix, profile the hot path, and re-run the same benchmark. Compare p50, p95, and p99, not only the average.
Expected state: The benchmark matches or beats the baseline and the fix is covered by a test.
6. Add a guardrail
Add a performance budget, a benchmark in CI, or a production alert tied to the affected percentile. Document the baseline and the load profile.
Expected state: The next regression of the same kind triggers before users report it.
Decision points
Should the team roll back or fix forward?
- Roll back immediately and investigate on the previous build
- Keep the new build and apply a targeted fix
Roll back when user impact is ongoing and the cause is not yet understood. Fix forward only when the fix is small, verified, and the rollback path is riskier.
Which metric should gate the release?
- Average response time
- p95 or p99 latency plus saturation
Use a high percentile and saturation. Averages hide the tail that users actually experience.
Expected outcome
The regression is explained by evidence, users are relieved by a reversible mitigation, and a guardrail detects the same class of regression.
Verification
Re-run the benchmark with the same load profile and compare percentiles; confirm the production alert fires on a simulated regression and that the rollback path still works.
Cleanup
Remove temporary capacity, test flags, and profiling overhead; keep the benchmark and baseline in version control.
A regression needs a baseline
“The app is slow” is not actionable. A regression is a change against a known baseline: p95 for GET /checkout rose from 180 ms to 640 ms after deploy 42, while error rate stayed flat. That statement can be tested. Start by capturing the metric, the comparison window, and the affected population.
hey -z 30s -c 50 https://staging.example.com/checkoutRun the same command against the previous build and the current build in the same environment. If the difference disappears when the load changes, the problem may be capacity or traffic mix rather than code. Production canary comparisons are often better than staging when the workload is hard to reproduce.
Narrow before you optimize
Correlate the start time with deploys, flags, dependency versions, and database changes. If several changes shipped together, use cohorts or git bisect in a controlled environment. A flame graph or trace can locate the slow span, but only after the regression is reproduced. Optimizing without a reproduction often improves a path that was not the problem.
Mitigate, then fix
Rollback is a valid diagnosis tool. It restores users and confirms that the change window contains the cause. Feature flags can disable a new path without a full deploy. If the regression is capacity-related, scaling may relieve users, but record it as a mitigation, not a fix.
Guard the percentile
Add a benchmark to CI or a production alert on the affected percentile. Compare p50, p95, and p99 with a fixed load profile. A performance budget that only checks the average will miss tail latency, and tail latency is what users and downstream timeouts experience.