Add a Circuit Breaker Around a Dependency

Wrap a flaky dependency with a closed/open/half-open circuit breaker, cap failures, allow one probe, and expose state for metrics.

~30 minresiliencenodejsreliability

Task

Stop calling a failing dependency after a threshold, fail fast to callers, and probe recovery without a thundering herd.

When to use this

A third-party API starts timing out. Retries and new requests keep hitting it, consuming connection pool capacity and delaying every caller.

Prerequisites

A Node.js service with a single client for the dependency
A metrics or logging path for breaker state changes
Timeouts on the underlying calls

Steps

1. Implement a breaker with a failure threshold, cooldown, and a single half-open probe.

export class CircuitBreaker {
  private state: 'closed' | 'open' | 'half-open' = 'closed';
  private failures = 0;
  private openedAt = 0;
  private probing = false;

  constructor(private threshold = 5, private cooldownMs = 30_000) {}

  async call<T>(operation: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.openedAt < this.cooldownMs) throw new Error('circuit open');
      this.state = 'half-open';
    }
    if (this.state === 'half-open') {
      if (this.probing) throw new Error('circuit half-open');
      this.probing = true;
    }
    try {
      const result = await operation();
      this.failures = 0;
      this.state = 'closed';
      return result;
    } catch (error) {
      this.failures += 1;
      if (this.state === 'half-open' || this.failures >= this.threshold) {
        this.state = 'open';
        this.openedAt = Date.now();
      }
      throw error;
    } finally {
      this.probing = false;
    }
  }
}

Expected output: After five failures the breaker opens; calls fail immediately until the cooldown, then one probe is allowed.

2. Wrap every call to the dependency through the breaker and keep the per-call timeout.

const breaker = new CircuitBreaker(5, 30_000);
const result = await breaker.call(() => fetchJson(url, { signal: AbortSignal.timeout(2000) }));

Expected output: A hung dependency is bounded by the timeout; repeated failures open the breaker.

3. Test closed, open, and recovery states with a fake operation that fails on demand.

node --test circuit-breaker.test.mjs

Expected output: The test observes fast failures while open and a successful close after the probe.

Troubleshooting

Problem

The breaker opens for business errors

Solution

Count only infrastructure failures (timeouts, 5xx, connection errors), not expected 4xx or validation errors.

Problem

A burst of probes hits the recovering dependency

Solution

Allow exactly one half-open probe and reject concurrent callers while it runs.

Limitations

A breaker is per process; multiple instances have independent state.
It complements timeouts and retries but does not repair the dependency.

Fail fast, then probe

A circuit breaker converts repeated slow failures into immediate local failures. The caller can serve a fallback or return a clear error without consuming a connection. The state machine has three states: closed (normal), open (fail fast), and half-open (one probe).

if (breaker.state === 'open') throw new Error('circuit open');

Tune the threshold

Set the failure threshold from the dependency’s normal behavior and the caller’s tolerance. Five consecutive failures may be right for a low-volume call and wrong for a high-volume endpoint where occasional failures are normal. Consider a rolling window instead of a strict consecutive count. Add jitter to the cooldown so multiple instances do not probe in lockstep.

Verify transitions

Write tests that force failures, observe the open state, advance time or inject a clock, and confirm exactly one probe is allowed. Assert that business errors do not open the breaker. Export state transitions and rejection counts as metrics; a breaker that opens silently makes the fallback path invisible.