Load Balancing Algorithms and When Each One Fails
Round robin, least connections, and consistent hashing spread traffic differently; the wrong algorithm creates hotspots that look like capacity problems.
Definition
A load balancing algorithm decides which backend receives each request or connection. Common families include round robin, weighted round robin, least connections, least response time, random with power of two choices, and consistent hashing.
Context and scope
Algorithm choice matters most when backends are heterogeneous or request costs vary widely; with identical, short requests the differences are usually small.
Common misconceptions
- Least connections is always best — it needs accurate counts and performs poorly with very short requests
- Round robin balances load — it balances request counts, not work
- Consistent hashing improves balance — it improves stability and can worsen balance
Anti-patterns
- Using round robin when request costs differ by orders of magnitude
- Rehashing every key when a single instance leaves the pool
- Adding weights that nobody recalibrates after a capacity change
Related concepts
Related examples
Authoritative source
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
Counting is not measuring
Round robin gives every backend the same number of requests. That is fair only if every request costs the same. When one endpoint triggers a report and another returns a cached record, equal counts produce wildly unequal CPU load. The result looks like a capacity problem but is really a distribution problem.
The common algorithms
- Round robin: simple, predictable, blind to cost.
- Weighted round robin: encodes known capacity differences, but weights rot.
- Least connections: adapts to slow backends, assuming connections reflect work.
- Least response time: reacts to latency, at the cost of measurement noise.
- Power of two choices: pick two at random, choose the less loaded; cheap and effective.
For example, a gateway fronting mixed read and write endpoints often does better with power of two choices than with least connections, because writes hold connections far longer than reads.
Consistent hashing’s trade
Consistent hashing keeps a key on the same backend as instances join and leave, which is essential for caches and sharded state. The trade is balance: a small number of virtual nodes can leave one backend with a disproportionate share. More virtual nodes smooth the distribution but increase memory and lookup cost.
Choosing
Start with the simplest algorithm that matches request cost, and measure per-backend load rather than request counts. Compared to a tuned algorithm, a misconfigured one exhausts connection pools long before backends saturate, so load test the balancer itself.
Weighing the trade-offs
Every algorithm optimizes something and ignores something else. Counting algorithms are cheap and stateless but blind to cost; measurement algorithms adapt but react slowly and can oscillate when latency is noisy; hashing algorithms stabilize affinity but sacrifice even distribution. None of them fixes a backend that is slow for everyone, and none replaces load shedding when demand exceeds total capacity.
Record the algorithm beside the pool configuration and review it whenever the workload changes shape; a choice that fit a read-heavy API often fails once writes dominate.