中文
devexamples
← Workflows

阅读中文版本

Runbook Quality Review Workflow

Test runbooks by having someone unfamiliar execute them, fix ambiguous steps and missing prerequisites, and keep them linked from alerts.

beginner~60 minoperationsrunbooksquality

Objective

Ensure runbooks actually work for the on-call engineer under pressure, not just look complete.

Environment

  • A runbook catalog
  • An engineer unfamiliar with the procedure to test

Steps

1. Have a person unfamiliar

Have a person unfamiliar with the system execute the runbook.

Expected state: Ambiguity is exposed in practice.

2. Fix missing prerequisites, exact commands,

Fix missing prerequisites, exact commands, and expected outputs.

Expected state: Steps are actionable and verifiable.

3. Add decision points and rollback guidance

Add decision points and rollback guidance.

Expected state: Complex situations are handled.

4. Link the relevant runbook from each

Link the relevant runbook from each alert and keep links current.

Expected state: Runbooks are reachable when needed.

Decision points

Document every step or assume knowledge?

  • Explicit steps
  • Assume familiarity

Explicit steps with exact commands; under pressure, assumed knowledge fails.

One runbook per symptom or per cause?

  • Per symptom
  • Per cause

Per symptom as alerts fire on symptoms; cross-link to cause-specific remediation.

Expected outcome

Runbooks proven executable by an unfamiliar engineer, with prerequisites, exact steps, and decision points.

Verification

Confirm the tester completed the procedure without undocumented help and that alert links resolve to the right runbook.

Cleanup

Record the review and update the runbook version.

Overview

Ensure runbooks actually work for the on-call engineer under pressure, not just look complete. Confirm the tester completed the procedure without undocumented help and that alert links resolve to the right runbook.

Steps

  1. Have a person unfamiliar with the system execute the runbook. Expected state: Ambiguity is exposed in practice.
  2. Fix missing prerequisites, exact commands, and expected outputs. Expected state: Steps are actionable and verifiable.
  3. Add decision points and rollback guidance. Expected state: Complex situations are handled.
  4. Link the relevant runbook from each alert and keep links current. Expected state: Runbooks are reachable when needed.
# a good runbook step has an exact command and an expected output
kubectl get pods -n prod | grep -v Running
# expected: no output means all pods are Running

Verification

  • Runbooks proven executable by an unfamiliar engineer, with prerequisites, exact steps, and decision points.

Decision points

  • Document every step or assume knowledge? Options: Explicit steps; Assume familiarity. Recommended: Explicit steps with exact commands; under pressure, assumed knowledge fails.
  • One runbook per symptom or per cause? Options: Per symptom; Per cause. Recommended: Per symptom as alerts fire on symptoms; cross-link to cause-specific remediation.