Add an Error-Budget Burn-Rate Alert

Alert on how fast the error budget is being consumed with multi-window burn rates, instead of waiting for a monthly SLO to be exhausted.

~35 minreliabilityobservabilityalerting

Task

Detect a severe reliability incident early while avoiding noisy alerts from normal background errors.

When to use this

A service has a 99.9% monthly SLO, but the team only notices a problem when the monthly budget is gone or customers complain.

Prerequisites

A reliable SLI recording rule
An SLO target and error-budget definition
An alerting system that supports PromQL or equivalent expressions

Steps

1. Compute the burn rate as observed error ratio divided by the allowed error ratio.

# Allowed error ratio = 1 - 0.999 = 0.001
(
  sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))
) / 0.001

Expected output: A burn rate of 1 consumes the budget exactly over the SLO window; 14.4 consumes it in about two days.

2. Use a fast and a slow window so short spikes and sustained burns are both caught, and require both to fire.

- alert: ApiErrorBudgetFastBurn
  expr: |
    (job:error_ratio5m > (14.4 * 0.001))
    and
    (job:error_ratio1h > (14.4 * 0.001))
  for: 2m
  labels:
    severity: page

Expected output: A severe incident pages quickly; a brief blip does not.

3. Add a slower ticket-level alert for a budget that will exhaust over days, and test both with injected failures.

amtool alert add test severity=page

Expected output: The fast alert fires for a large failure burst and the slow alert fires for a sustained leak.

Troubleshooting

Problem

Alerts fire on a single failed request

Solution

Require a minimum request volume or use a ratio over a window; a single error in ten requests is not a stable signal.

Problem

The alert never fires despite failures

Solution

Check the status label, the denominator, and whether the recording rule is evaluated. Verify with a known failing request.

Limitations

Burn-rate alerts depend on a correctly defined SLI; a bad SLI produces meaningless alerts.
Multi-window alerts require recording rules or a backend that supports the expressions.

Alert on speed, not just total

A monthly budget is a lagging indicator. Burn rate tells you how fast it is disappearing. A burn rate of 14.4 consumes a 30-day budget in roughly two days; a rate of 1 consumes it in 30 days. Alerting at a high burn rate catches incidents while there is still time to act.

burn rate = observed error ratio / allowed error ratio
allowed = 1 - SLO

Two windows

A five-minute window catches sudden failures but is noisy. A one-hour window catches sustained problems but reacts slowly. Requiring both to exceed the threshold reduces false pages while still detecting real incidents. Add a slower, lower-severity alert for a budget that will exhaust over days.

Test the alert

Inject a controlled failure rate and confirm the alert fires at the expected time. Test the recovery path and the silence behavior. An alert that has never fired is not known to work; an alert that fires on every deploy trains people to ignore it.