Run a Monthly SLO Review

Pull error budget burn from Prometheus, compare it with the SLO target, and decide whether to ship features or invest in reliability.

~30 minsloreliabilityprometheus

Task

Turn SLO data into a recurring decision instead of a dashboard nobody reads.

When to use this

The team tracks availability on a dashboard but never meets about it. The error budget is exhausted halfway through the quarter, and the discussion about slowing down happens only after a major outage.

Prerequisites

Prometheus with request counters labeled by status code
A documented SLO with a target and a measurement window
A recurring meeting with engineering and product owners

Steps

1. Record the error ratio with a rule so every review uses the same definition.

groups:
  - name: slo
    rules:
      - record: job:request_error_ratio:5m
        expr: |
          sum(rate(http_requests_total{job="payments-api",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="payments-api"}[5m]))

Expected output: The recording rule produces a stable series that dashboards and alerts can share.

2. Query the thirty-day availability and convert it into remaining error budget.

1 - (
  sum(increase(http_requests_total{job="payments-api",code=~"5.."}[30d]))
  /
  sum(increase(http_requests_total{job="payments-api"}[30d]))
)

Expected output: A value such as 0.9987, which is below a 99.9 percent target and means 87 percent of the budget is consumed.

3. Prepare a short agenda: budget consumed, top error sources, and the ship-versus-fix decision.

promtool query instant http://prometheus:9090 'job:request_error_ratio:5m{job="payments-api"}'

Expected output: The current error ratio is attached to the review notes alongside the thirty-day figure.

4. Record the decision, the owner, and the follow-up date in the SLO review log.

git add docs/slo-reviews/2026-10.md && git commit -m 'docs: record October SLO review and freeze decision'

Expected output: The review log states the budget status, the decision, and who owns each follow-up action.

Troubleshooting

Problem

The availability number looks better than user reports

Solution

Check that all ingress paths are counted and that 4xx client errors are excluded deliberately; missing a path hides real failures.

Problem

Low traffic makes the ratio jump wildly

Solution

Use a longer window or a request-count floor; a single failure on ten requests is not a meaningful SLO signal.

Limitations

An availability SLO does not capture latency or correctness; pair it with a latency objective.
A thirty-day window reacts slowly to sudden regressions; use burn-rate alerts for shorter horizons.

SLOs need a meeting, not a dashboard

An SLO defines the reliability users should expect and the error budget that allows risk. The value comes from the decision it enables: when the budget is healthy, ship faster; when it is exhausted, prioritize reliability work. Without a recurring review, the number is just decoration.

Measure one thing well

Start with availability: successful requests divided by total valid requests over thirty days. Exclude client errors deliberately and document the choice. A recording rule keeps the definition consistent across dashboards, alerts, and the review. If you cannot measure a path, say so; an incomplete measurement is worse than a narrower one.

1 - (
  sum(increase(http_requests_total{job="payments-api",code=~"5.."}[30d]))
  /
  sum(increase(http_requests_total{job="payments-api"}[30d]))
)

Run a fifteen-minute review

Cover three questions: how much budget was consumed, what consumed it, and what changes next. Attribute errors to deploys, dependencies, and traffic patterns. Decide explicitly whether to freeze risky launches or continue. Record the decision with an owner and a date so the next review can check whether it worked.

Connect to action

A burn-rate alert should page before the budget is gone, using a short window for fast burns and a long window for slow ones. When the review finds a recurring cause, convert it into a tracked reliability item with the same priority process as product work. Review the SLO target itself quarterly; a target that is always met may be too loose, and one that is never met may be unrealistic.