Runbook
A runbook is a tested, step-by-step operational procedure that lets an on-call engineer diagnose or remediate a known condition without rediscovering the system.
Definition
A runbook is a versioned operational document containing concrete steps, commands, expected outputs, and escalation paths for handling a specific alert, failure mode, or routine task.
Context and scope
Linked from alerts and used during on-call handoffs, incident mitigation, and repeatable maintenance work.
Common misconceptions
- A runbook is a wiki page written once and left unchanged
- Runbooks replace the need to understand the system
- Automation makes runbooks obsolete
Anti-patterns
- Storing exact commands only in one engineer's shell history
- Writing steps without expected output or a verification check
- Letting a runbook drift while alerts still link to it
Related concepts
Authoritative source
How it works
A runbook starts from a concrete trigger such as an alert name or task, states prerequisites and access needs, then lists numbered steps with exact commands, expected output, and verification. It names an owner, a last-tested date, and an escalation path. Where possible, steps become scripts, and the runbook links to them rather than duplicating them.
When to use it
Use runbooks for known failure modes, repeatable maintenance, and on-call procedures that must work at 03:00 under stress. They are less useful for novel incidents, where investigation matters more than a checklist. Compared with tribal knowledge, a runbook shortens time to mitigate and survives staff turnover.
Trade-offs
Runbooks rot as systems change, and a stale command can cause a second outage. Maintaining them competes with feature work, so link them from alerts and test them during quiet periods. Over-prescriptive runbooks suppress judgment; under-specified ones leave too much to rediscovery.
Example
A consumer-lag alert links to a runbook: confirm lag on the dashboard, scale consumers to six, verify lag is falling, and escalate to the platform team if it is not resolved within ten minutes. Each command includes expected output, so an engineer can spot a wrong cluster immediately.
Misconceptions
A runbook is not a wiki page written once and forgotten; it is versioned, tested operational knowledge. It does not replace understanding of the system, and automation does not make it obsolete, because someone must still diagnose when the script fails. Nor is every runbook a disaster plan.
Related concepts
Incident command assigns who executes; a postmortem feeds new runbooks and fixes; graceful shutdown is a procedure that often deserves one.