阅读中文版本

Build a Circuit Breaker Proxy

A proxy that wraps an upstream with a circuit breaker, tripping open on error-rate thresholds, half-opening to probe recovery, and closing when probes succeed.

circuit-breaker-proxy/
├── src/breaker.ts
├── src/proxy.ts
├── src/window.ts
├── tests/breaker.test.ts
└── package.json

window.push({ok:false, t:Date.now()}); const rate = failures/total;

if (rate > 0.5 && total > 20) state = "open";

if (state==="open" && now > trippedAt+cooldown) state = "half-open";

if (++consecutive >= 3) state = "closed";

Project overview

A proxy that forwards every request to an upstream that is failing is not protecting anything, it is just adding latency to an error. A circuit breaker watches the error rate and, when it crosses a threshold, stops forwarding and returns a fallback immediately. The upstream gets a chance to recover, and the caller gets a fast answer instead of a slow timeout.

The breaker is a state machine with three states. Closed is the normal state: requests pass through and the breaker counts failures in a sliding window. Open is the tripped state: requests are rejected immediately with a fallback, and a timer counts down to a probe. Half-open is the probe state: one request is allowed through as a test, and its result decides whether the breaker closes or re-trips.

The sliding window is what makes the trip decision responsive. A window that covers all history takes too long to notice a recovery; a window that covers only the last second flaps on a single failure. A time-bounded window of ten or thirty seconds reflects recent traffic and ignores the distant past.

The half-open probe is what makes the recovery safe. Opening the breaker to all traffic the moment the cooldown expires can flood a recovering upstream and re-trip it immediately. Letting one request through as a probe, and only opening to all traffic after consecutive probes succeed, gives the upstream a gradual ramp.

The value of this project is the state machine, not the proxy. Forwarding requests is trivial; deciding when to stop forwarding and when to start again, without flapping and without flooding a recovering service, is the part worth learning, because a breaker that flaps is worse than no breaker at all.

What you will build

The project delivers five capabilities, each driven by a test:

• Error-rate trip with sliding window
• Half-open probe with gradual recovery
• Closed on consecutive successes
• Per-upstream isolated state
• Fallback response on open

The proxy holds a per-upstream state machine: closed passes requests and counts failures in a sliding window; open rejects immediately with a fallback; half-open lets one probe through and transitions on its result. A timer moves open to half-open after a cooldown.

Architecture and data model

The sliding window is a ring buffer of samples, each a boolean and a timestamp. A sweep removes samples older than the window length before computing the rate, so the rate always reflects the recent past. The minimum sample count prevents a trip on the first failure, which would protect nothing and annoy callers.

The state machine transitions are guarded by the window. Closed to open happens when the rate crosses the threshold and the sample count meets the minimum. Open to half-open happens when the cooldown timer fires. Half-open to closed happens on consecutive successes; half-open to open happens on any failure, conservatively, because a failing probe means the upstream is not yet healthy.

The half-open state allows exactly one in-flight probe. Concurrent requests that arrive during the probe get the fallback, not the probe, because a probe is a test of the upstream, not a way to serve traffic. This limits the probe blast radius to one request.

The fallback is a static response configured per upstream, typically a 503 with a retry-after header. A dynamic fallback, such as serving from a stale cache, is a real improvement but out of scope, because it couples the breaker to a cache the breaker does not own.

Each upstream gets an isolated breaker, keyed by the upstream URL. A map of breakers lets the proxy front multiple upstreams without one failing upstream tripping another, because the states and windows are independent.

Prerequisites and setup

You need the following before you begin:

• Node.js 20 LTS or later
• An upstream service that may fail

The dependencies cover the core needs:

• undici (^6.19.0) — HTTP client to upstream
• pino (^9.4.0) — Structured logs

pnpm init
pnpm add undici
pnpm add pino
pnpm add -D typescript tsx vitest

Keep secrets and configuration in the environment, and validate them at boot so a misconfigured service fails loudly instead of silently dropping work.

Directory structure

circuit-breaker-proxy/ is organized so each concern is testable in isolation:

circuit-breaker-proxy/
├── src/breaker.ts
├── src/proxy.ts
├── src/window.ts
├── tests/breaker.test.ts
└── package.json

Phase 1: Sliding window

Track successes and failures in a time-bounded sliding window to compute the error rate.

window.push({ok:false, t:Date.now()}); const rate = failures/total;

Old samples fall off the window so the rate reflects recent traffic, not all history.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 2: Trip to open

When the error rate crosses the threshold with a minimum sample count, transition to open.

if (rate > 0.5 && total > 20) state = "open";

A minimum sample count prevents a trip on the first failure.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 3: Half-open probe

After a cooldown, move to half-open and let one request through as a probe.

if (state==="open" && now > trippedAt+cooldown) state = "half-open";

Only one probe at a time; concurrent requests in half-open get the fallback.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Phase 4: Close on recovery

On consecutive probe successes, transition to closed and reset the window.

if (++consecutive >= 3) state = "closed";

A single probe failure re-trips to open, conservatively.

The phase is driven by a test that asserts the expected behavior, so the implementation is correct before the next phase builds on it.

Testing

Drive the upstream to fail above the threshold and assert the breaker opens and returns fallbacks; advance the cooldown and assert a probe is allowed; drive the probe to succeed and assert the breaker closes after the consecutive count.

The tests drive a fake upstream that fails on command. The trip test sets the upstream to fail every request, sends enough to cross the threshold, and asserts the breaker opens and subsequent requests get the fallback without hitting the upstream.

The probe test advances the clock past the cooldown, sends one request, and asserts it hits the upstream as a probe. A concurrent request during the probe asserts it gets the fallback, proving the half-open state allows only one in-flight probe.

The close test drives the probe to succeed, sends enough probes to meet the consecutive count, and asserts the breaker closes and all subsequent requests hit the upstream. The re-trip test drives a probe to fail and asserts the breaker returns to open immediately.

The window test sends a burst of failures, then a burst of successes, and asserts the rate reflects only the recent successes after the old failures fall off the window. The isolation test fronts two upstreams, trips one, and asserts the other still passes requests.

The minimum-sample test sends one failure and asserts the breaker does not trip, proving the minimum count prevents a premature trip.

pnpm vitest run

Observability

Track the error-rate trip with sliding window rate, the circuit-breaker resource usage, and the error count as the three primary signals for the circuit-breaker-proxy system. A sudden change in any of these is the signal that something has shifted, not necessarily broken but worth a look before it becomes an incident. Chart the half-open probe with gradual recovery alongside the error-rate trip with sliding window so a correlation is visible at a glance, because a rising half-open probe with gradual recovery with a stable error-rate trip with sliding window means the system is doing more work per unit, not that it is failing.

Log every sliding window transition with the context that triggered it, because an operator who sees the symptom but not the cause will guess wrong. The log entry should carry the circuit-breaker identifier, the input that caused the transition, and the outcome, so the operator can reconstruct what happened without re-running it. A log without context is a line of text; a log with context is a diagnosis.

Set alerts on the derived metrics, not the raw counters. A rate of change is more actionable than a total, because a total grows monotonically and never tells you when the growth accelerated. Alert on the error-rate trip with sliding window error rate crossing a threshold, the circuit-breaker utilization exceeding a percentage, and the half-open probe with gradual recovery latency exceeding the SLO. Each alert should have a runbook link, so the operator who receives it at three in the morning knows what to do, not just what is wrong.

The circuit-breaker-proxy system should expose its metrics in a format the existing dashboard can scrape, so the observability is additive, not a new tool. A metric that nobody reads is noise, so before adding one, confirm it has a consumer. The dashboard has three panels: throughput, error rate, and resource utilization, each with a threshold line that is the SLO, so a glance tells an operator whether the system is healthy without reading numbers.

Deployment and operations

Run the proxy as a sidecar next to the upstream, or as a middleware in the caller. A sidecar keeps the breaker state close to the upstream and survives caller restarts; a middleware keeps the state in the caller and is simpler to deploy. The trade-off is state persistence, not correctness.

Configure the threshold, the window, the cooldown, and the consecutive count from observed upstream behavior, not defaults. A threshold of fifty percent is a starting point, but an upstream that fails at ten percent under load may need a lower threshold. The window should be long enough to smooth noise and short enough to notice a recovery.

Log every state transition with the rate and the sample count that triggered it. A breaker that trips and recovers silently is hard to operate, because the operator sees the symptoms, the fallbacks, but not the cause. The transitions are the cause.

Do not share a breaker across request classes that have different failure characteristics. A slow endpoint and a fast endpoint share a window, and a failing slow endpoint trips the breaker for the fast endpoint too. Per-endpoint breakers are more memory but more accurate.

Security considerations

Treat every input to the circuit-breaker-proxy system as untrusted, because the error-rate trip with sliding window path is the one place an external caller controls what the system processes. Validate the input shape before any circuit-breaker operation, and reject early with a clear error rather than letting a malformed input propagate into the sliding window phase where it causes a confusing failure that is hard to trace back to its source.

Keep secrets and configuration in the environment, never in the repository. The undici credentials, the signing keys, and the service endpoints are all environment variables, read once at boot and validated, so a misconfigured circuit-breaker-proxy fails loudly at startup instead of silently misbehaving at runtime. A system that starts with a bad configuration and runs anyway is a system that will produce wrong results for hours before anyone notices.

Rate-limit the error-rate trip with sliding window endpoint per caller, because an unbounded circuit-breaker operation is a resource leak as much as a security problem. A caller that can trigger the trip to open path without limit can exhaust the system, whether intentionally or not. The rate limit is a per-caller ceiling that costs legitimate callers nothing and stops the abuse before it reaches the sliding window logic.

Audit-log every state-changing operation with the caller identity, the input, and the timestamp. The audit log is not for debugging, it is for accountability: if something went wrong, the log says who did what and when. The circuit-breaker-proxy system writes the audit log to an append-only store that the system itself cannot modify, so a compromised system cannot rewrite its history.

Do not expose internal errors to the caller. A stack trace is a map of the system internals, and an attacker who collects them learns the circuit-breaker structure. Return a generic error with a correlation id, and log the full error internally with the same id, so an operator can trace it without leaking it.

Limitations and next steps

• The breaker is per-upstream, not per-request-class; a slow endpoint and a fast endpoint share a state.
• No bulkhead; the proxy does not cap concurrent in-flight requests.
• The fallback is static; dynamic fallback is out of scope.

The breaker is per-upstream, not per-request-class. All requests to an upstream share a state, a window, and a threshold. If one endpoint fails and another is healthy, the breaker trips for both. Per-endpoint breakers solve this but cost more memory and more configuration.

There is no bulkhead. The proxy does not cap concurrent in-flight requests to the upstream, so a slow upstream can exhaust the proxy connection pool without tripping the breaker, because the requests are pending, not failed. A bulkhead is a separate resilience pattern and out of scope.

The fallback is static. A dynamic fallback that serves a stale cache or a default response is more useful but couples the breaker to a cache or a default provider, which this build does not do. The static fallback is a 503 with a retry-after, which is correct and unhelpful, and that is an honest trade-off.