Add an SLO Dashboard

Define a request success SLI, record it with Prometheus rules, and chart the SLO target, current attainment, and remaining error budget.

~40 minobservabilityreliabilitymetrics

Task

Make service reliability measurable with a dashboard that shows the SLI, the SLO target, and the remaining error budget over a defined window.

When to use this

The team has latency and error dashboards but cannot answer whether the service is meeting its 99.9% monthly objective or how much budget remains.

Prerequisites

Request metrics with status codes and a stable route label
Prometheus or a compatible metrics backend
An agreed SLI definition and SLO target

Steps

1. Record the success ratio as an SLI over a short window, excluding invalid client requests.

groups:
  - name: slo
    rules:
      - record: job:sli_success_ratio5m
        expr: |
          sum(rate(http_requests_total{job="api",status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api",status!~"4.."}[5m]))

Expected output: The recording rule produces a ratio between 0 and 1 for the service.

2. Compute error budget remaining over the SLO window.

# SLO 99.9% over 30 days
(1 - (1 - avg_over_time(job:sli_success_ratio5m[30d])))
/ (1 - 0.999)

Expected output: The dashboard shows the fraction of the error budget still available.

3. Add panels for the SLO line, current attainment, burn rate, and the top error routes, then verify against real traffic.

curl -s 'http://prometheus:9090/api/v1/query?query=job:sli_success_ratio5m' | jq .

Expected output: The panel values match the query and move when failures are injected.

Troubleshooting

Problem

The SLI is always 1

Solution

The status label or route filter is wrong, or the metric is not incrementing. Check a known failing request.

Problem

The budget calculation drifts

Solution

The window or averaging method differs from the SLO definition. Use one recording rule and one window everywhere.

Limitations

A dashboard is not an alert; pair the SLO with burn-rate alerts.
The SLI only measures what the metric records; missing routes or status classes create blind spots.

One definition, one dashboard

An SLO dashboard is only useful if it uses the same SLI definition as the alerting and the error-budget policy. Record the SLI once, then derive attainment and budget from that rule. Different queries in different panels produce numbers that disagree during an incident.

sum(rate(http_requests_total{status!~"5.."}[5m]))
/
sum(rate(http_requests_total{status!~"4.."}[5m]))

Show the budget, not only the percentage

The current attainment percentage is hard to interpret late in a window. The remaining error budget answers the decision question: can we ship, or should we stabilize? Show the budget as a percentage and as a time or request allowance. Add the burn rate so a fast incident is visible before the monthly budget is exhausted.

Verify with failure injection

Send a controlled number of failing requests and confirm the SLI, budget, and panels move as expected. A dashboard that cannot detect an injected failure is not evidence. Keep the verification request id and the time window in the runbook.