Incident Postmortem Workflow
Run a blameless review within a week, reconstruct the timeline from alerts and logs, find the contributing factors, and track actions with owners and dates.
Objective
Turn an incident into a written record that prevents recurrence by finding contributing factors and tracking actions to completion.
Environment
- The incident timeline from alerts, chat, and logs
- The people who responded, in a blameless setting
- A tracking system for actions with owners and dates
Steps
1. Schedule within a week
Book the review within a week of the incident while memory is fresh, and set a blameless tone in the invite.
Expected state: A meeting on the calendar with the responders and a written reminder that the goal is learning, not blame.
2. Reconstruct the timeline
Merge alerts, chat messages, and log events into a single timeline with timestamps and the action taken at each step.
Expected state: A timeline that reads as a story, with the first alert, the response, and the recovery all visible.
3. Find contributing factors
Ask why at each step, and list every condition that made the incident possible, not only the last action.
Expected state: A list of contributing factors that includes tooling, process, and code, not only the person who pushed.
4. Track actions
For every factor, write an action with an owner and a date, and put it in the tracking system.
Expected state: Every action has an owner and a date, and the tracking system shows them as open until done.
Decision points
Public or internal?
- Internal only
- Public for external incidents
Public for external incidents, so customers see the learning; internal for the rest.
When are actions done?
- When the owner marks them
- When a monthly review confirms
A monthly review confirms, so an action does not quietly slip.
Expected outcome
A written postmortem with a timeline, contributing factors, and actions with owners and dates, all blameless.
Verification
Confirm the postmortem has a timeline, a list of contributing factors, and actions with owners and dates, and that the tone is blameless.
Cleanup
Publish the postmortem to the team, link it from the incident, and review open actions in a monthly meeting until they are done.
Blameless and prompt
Run the review within a week while memory is fresh. Set a blameless tone: the goal is to understand how the system failed, not who to punish. People who fear blame hide the details that prevent the next incident.
timeline:
10:01 alert: error_rate > 1%
10:02 on-call acknowledged
10:05 rollback started
10:08 error_rate < 0.1%Factors, not a last action
Ask why at every step. The last action is rarely the only cause; the tooling, the process, and the code all contributed. List every condition that made the incident possible.
Actions with owners and dates
For every factor, write an action with an owner and a date. An action without an owner is wishful thinking. Review open actions monthly until they are done.
Verification
The postmortem has a timeline, contributing factors, and actions with owners and dates, and the tone is blameless. Publish external incidents publicly so customers see the learning; keep the rest internal.