Adaptive Concurrency Limits
Adaptive concurrency limits tune the in-flight request cap to the observed goodput, so the system pushes as hard as it can without sliding into the latency cliff where backpressure collapses.
Definition
Adaptive concurrency limits are a feedback control technique that measures the recent latency and goodput and adjusts the maximum in-flight requests — additive increase when latency is stable, multiplicative decrease when latency degrades — so the system operates near its knee and backs off before the overload cliff.
Context and scope
Applied in service mesh proxies (Netflix concurrency limits, gRPC adaptive), client-side throttling, and any client that must self-limit without knowing the server capacity.
Common misconceptions
- A fixed limit is simpler and sufficient — a fixed limit is right for one load; adaptive tracks the load as the server and its neighbors change
- Adaptive limits are just AIMD congestion control — they are an application of AIMD to request concurrency; the novelty is the signal (latency gradient) and the scope (per dependency)
Anti-patterns
- Increasing the limit on every success without a decrease on latency rise, driving the system into the cliff and holding it there
- Sampling latency so coarsely that the gradient is noise, making the controller chase random variation
Related concepts
Authoritative source
https://netflixtechblog.com/performance-under-load-3e41fa625c3c
What it is
Adaptive concurrency limits are a feedback control technique that measures the recent latency and goodput and adjusts the maximum in-flight requests — additive increase when latency is stable, multiplicative decrease when latency degrades — so the system operates near its knee and backs off before the overload cliff. Applied in service mesh proxies (Netflix concurrency limits, gRPC adaptive), client-side throttling, and any client that must self-limit without knowing the server capacity. The central difficulty is that interleaving is nondeterministic: the set of possible executions grows super-exponentially with the number of operations, and testing samples a vanishing fraction.
How it works
Correctness is established by reasoning about happens-before relationships, not by running more tests. Synchronization primitives are chosen for the specific guarantee needed — mutual exclusion, publication, or ordering — and composed only with a documented proof of absence of deadlock. Bounded resources apply backpressure rather than unbounded queuing that hides overload until memory exhausts.
When to apply and pitfalls
Apply whenever multiple flows share mutable state or resources. The dominant pitfall is reasoning sequentially about concurrent code — “this can’t happen because A finishes before B” — when in fact the scheduler can interleave at any instruction boundary. The second is using a coarse lock for simplicity and turning parallelism back into serialization.