Runbook Test Automation
Tune SRE practice: review alert routing, audit dashboard coverage, and confirm every alert has a runbook and a clear owner.
Objective
Reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner.
Environment
- A working sre environment with access to the target system and its tooling
- Configuration and credentials for the sre workflow installed and verified
- A rollback or undo path documented before any change is applied
Steps
1. Inventory current alerts
Export every alerting rule and record which ones fired in the last 30 days with their frequency and owner.
Expected state: A complete alert inventory exists with firing frequency and ownership.
2. Classify and tune
Mark each alert as page, ticket, or noise; raise the threshold on noise; ensure page alerts have a runbook link.
Expected state: Every alert has a classification and page alerts have a runbook.
3. Verify routing
Send a test alert through each route and confirm it reaches the correct receiver at the correct severity.
Expected state: Test alerts land in the correct queue with the expected urgency.
Decision points
Which alerts should page a human?
- User-facing degradation or SLO burn
- Any metric above threshold
Page only on user-facing impact or SLO burn; route the rest to tickets to protect on-call focus.
Expected outcome
Reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner. The change is verified and a rollback path is confirmed.
Verification
Reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner. Compare before and after metrics and confirm the expected state is reached.
Cleanup
Archive the run artifacts, remove temporary credentials, and update the runbook with any changes.
Overview
Tune SRE practice: review alert routing, audit dashboard coverage, and confirm every alert has a runbook and a clear owner. The goal is to reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner.
amtool alert list --alertmanager.url=http://alertmanager:9093Steps
-
Inventory current alerts. Export every alerting rule and record which ones fired in the last 30 days with their frequency and owner. Expected: A complete alert inventory exists with firing frequency and ownership.
-
Classify and tune. Mark each alert as page, ticket, or noise; raise the threshold on noise; ensure page alerts have a runbook link. Expected: Every alert has a classification and page alerts have a runbook.
-
Verify routing. Send a test alert through each route and confirm it reaches the correct receiver at the correct severity. Expected: Test alerts land in the correct queue with the expected urgency.
Verification
Confirm the outcome by checking: reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner. Run the verification command and compare the result against the expected state. If any check fails, stop and investigate before continuing.
Decision points
- Which alerts should page a human?
- User-facing degradation or SLO burn
- Any metric above threshold
- Recommendation: Page only on user-facing impact or SLO burn; route the rest to tickets to protect on-call focus.