Connection Pool Exhaustion Workflow
Diagnose and resolve connection pool exhaustion by measuring pool utilization, identifying leaks, tuning the pool size, and adding a circuit breaker on the downstream.
Objective
Restore service availability when the connection pool is exhausted, by finding the cause (leak, undersized pool, or slow downstream) and applying the right fix.
Environment
- A service with a connection pool and health metrics
- Access to pool utilization and wait-time metrics
- The downstream the pool connects to
Steps
1. Measure pool state
Check the pool utilization, the wait time for a connection, and the downstream latency to identify which is the bottleneck.
Expected state: A diagnosis: high utilization with low wait means the pool is undersized; high wait with a leak means connections are not returned.
2. Identify leaks
If utilization is high with low traffic, trace connection acquisition and release to find a path that does not return the connection.
Expected state: A leak is found or ruled out.
3. Tune the pool or the downstream
If the pool is undersized, raise the limit; if the downstream is slow, add a timeout and a circuit breaker so a slow downstream does not hold connections.
Expected state: The pool has capacity or the downstream is bounded.
4. Verify recovery
Run the traffic that caused the exhaustion and assert the pool utilization stays below 80% and no request times out waiting for a connection.
Expected state: The pool is no longer exhausted under the same load.
Decision points
Raise pool size or fix downstream?
- Raise the pool size
- Fix the downstream
Fix the downstream first; a bigger pool on a slow downstream just delays the exhaustion.
Leak fix or pool restart?
- Fix the leak in code
- Restart the pool as a bandage
Fix the leak; a restart is a bandage that returns if the leak is not patched.
Expected outcome
The connection pool handles the traffic that caused exhaustion without timing out, and the root cause is fixed, not just the symptom.
Verification
Run the exhaustion-triggering traffic for 10 minutes and confirm pool utilization stays below 80%, wait time is near zero, and no request fails with a pool timeout.
Cleanup
Keep the pool tuning, the timeout, and the circuit breaker; remove the diagnostic traces.
Measure the pool state
Connection pool exhaustion has three causes: the pool is undersized for the traffic, connections are leaked and not returned, or the downstream is slow and holds connections too long. The metrics tell which.
metrics pool.utilization pool.wait_time downstream.latencyHigh utilization with low wait means the pool is undersized. High wait with low traffic means connections are leaked. High wait with high downstream latency means the downstream is the bottleneck.
Identify leaks
A leak is a code path that acquires a connection and does not return it. Trace the acquisition and release, and look for a path where the release is missing, typically in an error handler that throws before the finally block.
grep -n "pool.acquire" src/ && grep -n "pool.release" src/Tune the pool or bound the downstream
If the pool is undersized, raise the limit to match the traffic. If the downstream is slow, add a query timeout and a circuit breaker so a slow downstream does not hold a connection indefinitely.
pool.max = 100; query.timeout = 5s; breaker.threshold = 50%Verification
The traffic that caused the exhaustion now runs for 10 minutes with pool utilization below 80%, wait time near zero, and no pool timeout. The root cause is fixed: the leak is patched, the pool is sized, or the downstream is bounded.