Circuit Breaker Tuning
Circuit breaker tuning sets the failure threshold, open duration, and half-open probe count so the breaker trips on real outages, not on a blip, and recovers without hammering the recovering service.
Definition
Circuit breaker tuning is the configuration of the failure threshold that opens the circuit, the time to wait before half-open probing, and the number of half-open requests that determine recovery, so that the breaker opens on a sustained outage rather than a transient blip and recovers gradually without overwhelming the still-recovering dependency.
Context and scope
Applied in service-to-service calls, database connections, and any dependency whose failure is detected by error rate and whose recovery must be probed.
Common misconceptions
- A lower threshold is safer — it is safer against the outage; it is more likely to trip on a blip and cause unnecessary degradation, so the threshold is a trade-off
- Half-open is just retry — half-open is a controlled retry with a recovery test; an uncontrolled retry is a thundering herd
Anti-patterns
- Setting the threshold so low that a single error trips the breaker, degrading on every transient blip
- Setting the open duration so short that the breaker recovers, fails, and trips again, oscillating and never stabilizing
Related concepts
Authoritative source
What it is
Circuit breaker tuning is the configuration of the failure threshold that opens the circuit, the time to wait before half-open probing, and the number of half-open requests that determine recovery, so that the breaker opens on a sustained outage rather than a transient blip and recovers gradually without overwhelming the still-recovering dependency. Applied in service-to-service calls, database connections, and any dependency whose failure is detected by error rate and whose recovery must be probed. Reliability is a probabilistic property of a system under load, not a feature you add — it is the result of how failures are designed for, detected, and recovered from.
How it works
Failure is assumed; the design question is the blast radius of each failure mode. Redundancy eliminates single points of failure but introduces consistency and split-brain risks that must be solved explicitly. Error budgets quantify the reliability the system can afford, converting “we should be more reliable” into “we have spent this month’s budget and must choose between features and fixes.” Recovery is rehearsed, not theorized.
When to invest and pitfalls
Invest when downtime has a measurable cost — revenue, trust, or safety. The pitfall is adding redundancy without solving the consistency problem it creates — two copies that disagree are worse than one copy that is wrong. Another is chasing five nines without measuring where the downtime actually comes from, investing in the rare failure while ignoring the common one.