Adaptive Concurrency Limit Workflow
Tune a concurrency limit adaptively from observed latency, using additive increase and multiplicative decrease to find the sweet spot without manual setting.
Objective
Find the maximum concurrency that keeps latency under the SLO by adapting the limit from real-time measurements, not a static guess.
Environment
- A service with a concurrency limiter and latency metrics
- A load generator that can vary the offered concurrency
- An SLO on p99 latency
Steps
1. Measure at current limit
Apply the current concurrency limit, run the load generator, and record the p99 latency and the throughput.
Expected state: A measurement pair of p99 latency and throughput at the current limit.
2. Adjust the limit
If latency is under the SLO, increase the limit additively; if over, decrease it multiplicatively to back off fast.
Expected state: A new limit that moves toward the sweet spot without overshooting.
3. Confirm convergence
Repeat the measure-adjust cycle until the limit stabilizes within a band for several rounds.
Expected state: The limit oscillates in a narrow band around the sweet spot, not a single value.
4. Lock and monitor
Pin the converged limit as the default and add a dashboard alert if latency crosses the SLO, so a regression triggers re-adaptation.
Expected state: The limit is set and the alert watches for drift.
Decision points
Additive increase step size?
- Small (1) for stability
- Large (10) for speed
Small for production, large for initial tuning.
Multiplicative decrease factor?
- 0.5 for fast back-off
- 0.9 for gentle back-off
0.5, because overshooting latency is worse than undershooting throughput.
Expected outcome
The concurrency limit converges to a value that maximizes throughput while keeping p99 latency under the SLO, and a dashboard alert catches regressions.
Verification
Run the load generator at the converged limit and confirm p99 latency is under the SLO across 1000 requests, and that the limit did not change in the last 5 rounds.
Cleanup
Keep the converged limit and the alert; remove the load-generator artifacts and record the converged value in the runbook.
Measure before you adjust
A concurrency limit that is too high degrades latency; one that is too low wastes capacity. The right limit is where throughput stops rising and latency starts rising, and that point shifts with the load and the downstream, so a static guess is wrong eventually.
loadtest --concurrency $LIMIT --duration 30s --metric p99Additive increase, multiplicative decrease
When latency is under the SLO, increase the limit by a small additive step, because the sweet spot is nearby and a big jump overshoots. When latency is over the SLO, decrease the limit by a multiplicative factor, because a latency violation is urgent and a fast back-off recovers sooner.
if p99 < SLO: LIMIT=$((LIMIT + 1)); else LIMIT=$((LIMIT / 2)); fiConfirm convergence
The limit does not converge to a single value; it oscillates in a band around the sweet spot. Confirm convergence by checking the limit stayed within a band for several rounds, then pin the midpoint as the default.
Verification
The converged limit keeps p99 latency under the SLO across 1000 requests, and the limit did not change in the last 5 rounds. The dashboard alert watches for latency regressions that would trigger re-adaptation.