Error Budget Burn Response Workflow

Detect budget burn rate against SLOs, respond by throttling risky change or reallocating reliability work, and report burn transparently.

advanced~30 minsresloreliability

Objective

Turn SLO error budgets into an operational signal that automatically throttles feature velocity when reliability spending exceeds income.

Environment

Defined SLOs with error budgets per service
Burn-rate alerts configured (fast and slow windows)
Authority to gate deploys or freeze features on budget state

Steps

1. Set the burn-rate alerts

Configure multi-window burn alerts: fast burn (2% of budget in an hour) pages immediately, slow burn (10% over days) files for planning.

Expected state: Both alert classes exist with documented thresholds tied to the budget math.

2. Classify the burn source

When burn triggers, attribute it to change-induced regressions, capacity shortfalls, or dependency failures, since each has a different response lever.

Expected state: The burn incident names its source class with supporting deploy or capacity evidence.

3. Apply the matching throttle

Change-induced burn gates deploys until the regression is fixed; capacity burn triggers scaling; dependency burn invokes the dependency playbook.

Expected state: The response uses the lever matching the cause instead of universally freezing.

4. Report burn transparently

Publish the budget balance and burn curve to stakeholders, with what reliability work the remaining budget is being spent on.

Expected state: The budget state is visible without asking engineering.

5. Reset and review at the window boundary

At the budget window end, review burn sources and the throttling decisions, adjusting SLOs or capacity plans if burn was chronic and legitimate.

Expected state: Window reviews produce SLO or capacity adjustments, not just resets.

Decision points

Hard freeze or throttle on budget exhaustion?

Full feature freeze
Risk-tiered throttle

Risk-tiered: low-risk fixes continue, risky changes queue — a full freeze punishes the fix that restores budget.

What if the SLO is simply wrong?

Enforce anyway
Emergency SLO review

Emergency review when burn is chronic and customers are not complaining; enforcing an SLO nobody needs burns the budget and the team.

Expected outcome

Error budgets that operate as real policy: burn slows change when reliability is overspent, and window reviews adjust the underlying SLOs rather than perpetually fighting them.

Verification

A deliberately induced fast burn pages within the alert window; the deploy gate engages for change-classified burn and disengages on fix; budget dashboards are stakeholder-visible.

A budget nobody enforces is a decoration

Error budgets exist to arbitrate velocity versus reliability, but only if burning them has consequences. Burn-rate alerts tied to matching throttles convert the budget from a dashboard ornament into operating policy.

SLO: 99.9% availability  (budget 43.2 min / 30d)
alerts:
  fast burn: >2% budget in 1h    -> page
  slow burn: >10% budget in 3d    -> planning ticket

14:20Z fast burn page (14.1% err = 25x budget rate)
  source: v1.8.4 regression (deploy 14:02)
  throttle: deploy gate ON (risk-tiered)
    -> fixes proceed, risky changes queue
  fix + verify 15:31Z, gate OFF

window review: 3rd chronic capacity burn
  -> capacity plan +200ms headroom, not another freeze

Throttle the right class of change

Freezing everything when the budget bleeds punishes the people fixing it: their changes are the restore path. Risk-tiered throttling keeps fixes flowing while queuing the risky launches, which is the policy the budget was meant to encode.

Chronic burn means the SLO or the plan is wrong

If every window ends exhausted and customers are content, the SLO is stricter than user needs; if capacity burns the budget monthly, the capacity plan is fiction. Window reviews are where the system gets corrected, not just refilled.