← Workflows

阅读中文版本

Blameless Postmortem Workflow

Reconstruct an incident timeline, identify contributing system causes, assign verifiable action items, and review completion without blaming individuals.

intermediate~60 minoperationspostmortemreliability

Objective

Turn an incident into durable improvements by analyzing system conditions and tracking preventive actions to completion.

Environment

  • Incident timeline, logs, dashboards, and chat records
  • A facilitator who was not the primary responder
  • An action tracker with owners and due dates
  • A review process that revisits completed actions

Steps

1. Gather evidence

Collect timestamps, alerts, deploy records, configuration changes, and customer impact. Preserve links before retention windows expire.

Expected state: A factual timeline exists with sources for each event.

2. Map contributing causes

Identify technical, process, and organizational conditions that made the failure possible. Distinguish triggers from underlying causes.

Expected state: The analysis explains why the system allowed the failure, not who made a mistake.

3. Write the report

Document impact, timeline, detection, response, contributing causes, and what went well. Keep language specific and evidence-based.

Expected state: A reviewer can understand the incident without being in the room.

4. Define verifiable actions

Assign each action an owner, due date, and verification method. Prefer changes that prevent recurrence or shorten detection and recovery.

Expected state: Actions are concrete, tracked, and testable.

5. Review and close

Revisit actions at a scheduled review, verify they were completed and effective, and record any remaining risk.

Expected state: Completion is verified, not assumed from a ticket status.

Decision points

Should the postmortem be public?

  • Internal only
  • Published with sensitive details removed

Share internally by default; publish externally when it helps customers and does not expose security details.

How many actions should be created?

  • A small number of high-impact actions
  • A long list covering every observation

Prioritize a few verifiable actions with owners; a long unowned list rarely gets completed.

Expected outcome

A shared understanding of the incident, a prioritized set of preventive actions, and evidence that those actions were completed and tested.

Verification

Confirm the report has a sourced timeline, every action has an owner and due date, completed actions were verified, and at least one failure mode was retested where feasible.

Cleanup

Archive the report and evidence, link it from the incident record, and add recurring themes to the reliability backlog.

Blameless does not mean accountless

A blameless postmortem assumes people acted reasonably given the information and tools available. It focuses on system conditions: missing safeguards, unclear runbooks, unsafe defaults, and gaps in detection.

Timeline:     14:02 alert -> 14:05 declared -> 14:11 rollback -> 14:20 recovered
Impact:       3.2% of checkout requests failed for 18 minutes
Contributing: deploy lacked a canary gate; rollback was undocumented
Actions:      add canary gate (owner, date); document rollback (owner, date)

Analysis

  1. Build a sourced timeline.
  2. Separate trigger from contributing causes.
  3. Identify what went well and what slowed recovery.
  4. Write specific, evidence-based findings.

Verification

  • The timeline has timestamps and sources.
  • Every action has an owner, due date, and verification method.
  • Completed actions are reviewed and tested.
  • Recurring themes are added to the reliability backlog.
  • The report is linked from the incident record.

Common failure

Creating a long list of vague actions such as “improve monitoring” with no owner. Prefer a small number of concrete changes that can be verified, and schedule a review to confirm they happened.