Service Outage Triage Workflow

Establish severity from user impact, split diagnosis from communication, and drive restoration through hypothesis testing with a time box.

intermediate~20 minincidentsrereliability

Objective

Restore service through structured triage: severity from impact, parallel investigation tracks, and restoration hypotheses tested on a clock.

Environment

Monitoring with user-facing signals (error rate, availability)
An on-call rotation with a defined incident commander role
A status page and communication channel

Steps

1. Declare and grade the outage

Confirm user impact from external signals (availability, error rate, support volume), assign severity, and name an incident commander within minutes.

Expected state: An incident exists with a severity label, a commander, and a timestamped record.

2. Split diagnosis from communication

Separate the diagnosis track (what is broken, since when, what changed) from the communication track (status page updates every N minutes), staffed independently.

Expected state: The status page updates on schedule while diagnosis proceeds without meeting overhead.

3. Stabilize before root-causing

Attempt restoration levers first (rollback, failover, feature flag off, load shedding), each as a time-boxed hypothesis with an expected signal outcome.

Expected state: Each lever is tried with a stated hypothesis and evaluated against its expected signal change.

4. Widen the blast-radius check

While restoring, verify which downstream and upstream services are affected to prevent cascading action (e.g., failover under dependency failure).

Expected state: A dependency map of the incident exists before large architectural actions.

5. Confirm recovery and stand down

Verify user-facing signals return to baseline for a sustained window, stand down the incident, and schedule the postmortem.

Expected state: Signals hold at baseline, the incident record is complete, and the postmortem is calendared.

Expected outcome

Outages handled with fast severity assignment, parallel diagnosis and communication, and restoration driven by tested hypotheses rather than panic rotation.

Verification

The incident record shows severity within minutes, status updates on cadence, each restoration lever with hypothesis and outcome, and a recovery verification window.

Restoration first, explanation later

During an outage the only product is restored service. Grading severity from user impact, splitting diagnosis from communication, and trying restoration levers as time-boxed hypotheses keeps the response moving when everyone is stressed.

INC-2901  sev1  checkout unavailable  14:02Z
commander: oncall-lead    comms: support-lead

levers (time-boxed 10 min each):
  14:04 rollback v1.8.1 -> v1.8.0   err 78% -> 76%  (no)
  14:16 flag coupons=off            err 76% -> 31%  (partial)
  14:24 failover db-primary         err 31% -> 0.4%  (yes)

status page: 14:05 / 14:20 / 14:35 / 14:50 (fixed cadence)
recovery verified 14:24-14:54 (30 min sustained)
standdown 14:55, postmortem scheduled 10-23

Levers are hypotheses, not rituals

Rollback, failover, flag-off: each lever should state what signal it should move if it is the right one. That discipline prevents the worst incident pattern — rotating through every lever with no observation window and no learning between attempts.

Communication on a metronome

Users forgive outages; they do not forgive silence. A status page cadence held by a dedicated person (every 15 minutes, even to say nothing new) is the difference between a managed incident and a trust incident.