← 中文资源

简体中文版本

运行手册测试自动化

调优 SRE 实践:审查告警路由,审计仪表盘覆盖率,并确认每条告警都有对应运行手册与明确负责人。

阅读英文原文

概述

调优 SRE 实践:审查告警路由,审计仪表盘覆盖率,并确认每条告警都有对应运行手册与明确负责人。本工作流属于站点可靠性领域,目标是reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner.以下步骤将指导你完成关键操作,并在每一步设置可验证的中间状态,确保最终结果可复现、可审计、可回滚。

步骤详解

  1. Inventory current alerts。 Export every alerting rule and record which ones fired in the last 30 days with their frequency and owner.预期状态:A complete alert inventory exists with firing frequency and ownership.

  2. Classify and tune。 Mark each alert as page, ticket, or noise; raise the threshold on noise; ensure page alerts have a runbook link.预期状态:Every alert has a classification and page alerts have a runbook.

  3. Verify routing。 Send a test alert through each route and confirm it reaches the correct receiver at the correct severity.预期状态:Test alerts land in the correct queue with the expected urgency.

每一步都包含明确的动作与预期状态。执行时请记录实际输出,与预期状态逐一对照。若某一步未达预期,应停下排查而非继续推进,避免问题向后传递放大。

验证与后续

验证方法:reduce alert noise and ensure every page-worthy alert is actionable with a linked runbook and owner.完成后,将本次执行的命令、输出与参数归档,并更新团队运行手册。若用于生产环境,请先在预发环境完整演练一次,并确认团队熟悉回滚路径与应急联系人。本工作流难度级别为高级(advanced),预计耗时约40分钟。该流程已通过结构化校验,步骤与决策点均与英文版本严格一致,可在本地按工作流步骤复现并核对结果。

代码示例

amtool alert list --alertmanager.url=http://alertmanager:9093