Incident Response Runbook
Detect, declare, mitigate, communicate, and review an incident with explicit roles and evidence, prioritizing restoration over root-cause analysis during the outage.
Objective
Restore service safely during an incident while keeping a timeline and evidence that supports a blameless postmortem.
Environment
- Alerting and on-call rotation
- Dashboards for SLOs, errors, latency, and saturation
- A communication channel and status page
- Rollback and feature-flag controls
Steps
1. Acknowledge and declare
Acknowledge the alert, assess user impact, and declare an incident with a severity level. Assign an incident commander and a communications lead.
Expected state: A single incident channel, severity, and owner exist within minutes.
2. Stabilize before diagnosing
Stop the bleeding with a rollback, feature-flag kill switch, traffic shift, or dependency failover. Record the time and action.
Expected state: User impact stops increasing even if the root cause is still unknown.
3. Communicate on a cadence
Post concise updates at a fixed interval with impact, current mitigation, and next update time. Avoid speculation.
Expected state: Stakeholders know what is affected and when the next update will arrive.
4. Diagnose with evidence
Compare the timeline against recent deploys, configuration changes, dependency health, and capacity metrics. Preserve logs and dashboards before they rotate.
Expected state: Hypotheses are tied to observed evidence, not guesses.
5. Resolve and monitor
Confirm SLOs return to normal, keep heightened monitoring for a defined period, and close the incident only after stability.
Expected state: Error and latency budgets are back within target and no new alerts fire.
6. Run a blameless review
Schedule a postmortem, reconstruct the timeline, identify contributing causes, and assign owners and due dates for action items.
Expected state: Action items are tracked and verified in later reviews.
Decision points
Should we roll back or roll forward?
- Roll back to the last known good version
- Roll forward with a fix
Roll back first when the change is recent and rollback is safe; roll forward only when rollback would be slower or riskier.
When should the incident be declared?
- As soon as user impact is plausible
- After root cause is confirmed
Declare early; coordination and communication are cheaper than a late response.
Expected outcome
Service is restored with minimal user impact, communications are timely and accurate, and the incident produces concrete preventive actions.
Verification
Check that SLO dashboards are back within budget, the status page reflects resolution, the timeline has timestamps for detection, mitigation, and recovery, and every action item has an owner and due date.
Cleanup
Close the incident channel, archive evidence links, and remove temporary traffic shifts or feature flags after stability is confirmed.
Roles and cadence
An incident needs a commander who coordinates, a communications lead who updates stakeholders, and responders who investigate. The commander does not debug; they keep the response organized and decide when to escalate.
[detect] -> [declare] -> [mitigate] -> [communicate] -> [verify] -> [review]
| |
assign roles rollback/flagTimeline discipline
Record timestamps for detection, declaration, each mitigation attempt, and recovery. Preserve links to dashboards and logs before retention windows roll over. During the incident, prioritize restoring service; root-cause analysis belongs in the postmortem.
Verification
- Error and latency SLOs return to target.
- The status page and incident channel reflect resolution.
- The timeline explains what happened without blaming individuals.
- Action items have owners, due dates, and a verification method.
After the incident
Run a blameless review focused on systems and safeguards: missing alerts, unsafe defaults, insufficient testing, unclear ownership. Track action items to completion and re-test the failure mode where feasible.