Add a Blameless Postmortem Template

Ship a postmortem template with timeline, contributing factors, and owned action items, and enforce required sections in CI.

~30 minincident-responsedocumentationprocess

Task

Make incident write-ups consistent and actionable without turning them into blame exercises.

When to use this

Postmortems are written inconsistently: some have timelines, some only a paragraph, and action items rarely have owners. Two quarters later the same class of failure recurs and nobody can find the earlier analysis.

Prerequisites

A Git repository for engineering documents
Markdown and a CI job that can run shell checks
An issue tracker for action items

Steps

1. Add a template with fixed sections for impact, timeline, contributing factors, and action items.

# Incident: payments API latency

## Summary
One paragraph describing what users experienced.

## Impact
- Duration:
- Affected users:
- Error budget consumed:

## Timeline (UTC)
| Time | Event |

## Contributing factors
List system conditions, not people.

## What went well

## Action items
| Action | Owner | Due | Ticket |

Expected output: The template is committed at docs/postmortems/TEMPLATE.md and linked from the incident runbook.

2. Create the postmortem file during the incident, not weeks later, and fill the timeline from the chat and alert history.

cp docs/postmortems/TEMPLATE.md docs/postmortems/2026-10-21-payments-latency.md

Expected output: A dated file exists while the incident is fresh, with at least the start time and detection time recorded.

3. Add a CI check that requires the key headings and at least one action item row.

#!/usr/bin/env bash
set -euo pipefail

for file in docs/postmortems/2026-*.md; do
  grep -q '^## Timeline' "$file"
  grep -q '^## Action items' "$file"
  grep -q '^| Action | Owner |' "$file"
  grep -Eq '^\| [^|]+ \| [^|]+ \| 2026-' "$file"
done

Expected output: A postmortem missing a timeline or an owned action item fails CI with the file name.

4. Review action items weekly and close the postmortem only when every item has a ticket and an owner.

gh issue list --label postmortem-action --state open

Expected output: Open action items appear in the weekly operations review until they are completed or explicitly dropped.

Troubleshooting

Problem

Postmortems are written long after the incident

Solution

Open the file during the incident and record timestamps live; a rough timeline written immediately beats a polished one written from memory later.

Problem

Action items never get done

Solution

Require an owner, a due date, and a ticket for every item, and review them in a recurring meeting with the same visibility as other work.

Limitations

The CI check validates structure, not quality; a filled template can still lack insight.
Small incidents may not need a full postmortem; define a severity threshold for when the template is required.

Blameless does not mean actionless

A blameless postmortem focuses on system conditions rather than individual mistakes, because blame suppresses the information needed to prevent recurrence. That does not mean a soft document. It means a precise one: what happened, which defenses failed, and what concrete change reduces the chance or impact next time.

Structure forces completeness

A template with impact, timeline, contributing factors, and action items makes omissions visible. The timeline should include detection and response, not just the trigger, because slow detection is often the largest contributor. Contributing factors should describe conditions such as a missing timeout or an untested rollback, not a person’s lapse.

## Timeline (UTC)
| Time | Event |
| 09:12 | Deploy of payments-api 2.4.1 |
| 09:18 | p99 latency alert fires |

Enforce the minimum

A lightweight CI check that required headings exist and at least one action item has an owner and due date prevents the most common failure: a document that looks complete but commits to nothing. The check should not judge prose. Review quality in a weekly meeting where owners discuss progress.

Close the loop

Every action item needs a ticket in the normal backlog, not a separate list that decays. Review open items weekly, and mark the postmortem as closed only when each item is done, scheduled, or explicitly accepted as a risk. When a similar incident recurs, link the new postmortem to the old one and ask why the earlier action did not prevent it.