Isolate Downstreams with Bulkhead Pools

Give each downstream dependency its own bounded connection or concurrency pool so one slow service cannot consume every shared slot.

~35 minresilienceconnection-pooldatabases

Task

Stop a single slow dependency from starving unrelated requests by partitioning pools and failing fast when a partition is saturated.

When to use this

A service shares one database pool and one HTTP agent across payments, search, and reporting. When reporting slows down, connections are exhausted and payment requests fail even though the payment database is healthy.

Prerequisites

A connection or concurrency pool library such as node-postgres or HikariCP
Metrics for pool active, idle, and waiting counts
A latency injection tool such as Toxiproxy for testing

Steps

1. Create one bounded pool per dependency with an explicit wait timeout instead of one global pool.

import pg from 'pg';

export const paymentsPool = new pg.Pool({
  connectionString: process.env.PAYMENTS_DATABASE_URL,
  max: 8,
  connectionTimeoutMillis: 500,
  idleTimeoutMillis: 30_000,
});

export const reportsPool = new pg.Pool({
  connectionString: process.env.REPORTS_DATABASE_URL,
  max: 4,
  connectionTimeoutMillis: 500,
  idleTimeoutMillis: 30_000,
});

Expected output: Reporting traffic can occupy at most four connections and never blocks payment clients waiting for a slot.

2. Export per-pool saturation metrics and alert when clients wait for a slot.

setInterval(() => {
  for (const [name, pool] of Object.entries({ paymentsPool, reportsPool })) {
    poolMetrics.gauge('db_pool_total', { pool: name }, pool.totalCount);
    poolMetrics.gauge('db_pool_idle', { pool: name }, pool.idleCount);
    poolMetrics.gauge('db_pool_waiting', { pool: name }, pool.waitingCount);
  }
}, 5_000);

Expected output: Dashboards show separate saturation curves, and db_pool_waiting above zero for one minute triggers an alert.

3. Inject two seconds of latency into the reporting database only and compare endpoint behavior.

toxiproxy-cli toxic add reports-db -t latency -a latency=2000

Expected output: Reporting requests slow down or fail fast after 500 ms, while payment p95 stays unchanged.

Troubleshooting

Problem

Total pool connections exceed the database max_connections

Solution

Sum the maximums across every pool and replica, then keep the total below 80 percent of max_connections to leave room for admin sessions.

Problem

Clients leak connections and the pool slowly drains

Solution

Always release in a finally block or use a pool-aware transaction helper; monitor the difference between checked-out and idle counts.

Limitations

Separate pools increase total connection count and require per-dependency sizing.
A static split can waste capacity when traffic shifts between dependencies.

The shared-fate problem

One pool for every dependency couples their failure modes. A slow reporting query holds connections until the pool is empty, and healthy payment traffic queues behind it. A bulkhead removes that coupling by giving each dependency a fixed capacity and a bounded wait. When a partition is full, callers fail fast or degrade instead of consuming the whole process.

Size each partition

Set max from the dependency’s own capacity, not from total traffic. A read replica handling reports may tolerate four connections; the primary handling payments may need eight. Keep connectionTimeoutMillis short, around 500 ms, so a saturated pool returns an error quickly. Include the pool name in metrics and logs; without it, saturation is invisible.

const reportsPool = new Pool({
  connectionString: process.env.REPORTS_DATABASE_URL,
  max: 4,
  connectionTimeoutMillis: 500,
});

Prove isolation

Use Toxiproxy to add latency to one dependency and run a mixed workload. Payment latency should remain flat while reporting degrades in a predictable way. If payment latency also rises, some path still shares a pool, an HTTP agent, or a thread pool. Search for shared singletons and duplicate the boundary.

Plan for exhaustion

A bulkhead changes the failure mode from global starvation to a localized error. Pair it with a fallback: serve a cached report, return 503 with Retry-After, or queue the work. Review pool sizes quarterly against traffic; a partition that is always saturated is a capacity signal, not a reason to remove the isolation.