Run a Tabletop Exercise for a Critical Incident
Rehearse a realistic outage with responders and a facilitator, inject pressure over ninety minutes, and convert observations into owned, dated remediation work.
Objective
Exercise the incident response process with the people who would actually respond, surface decision and communication gaps, and turn the findings into owned actions.
Environment
- A facilitator who is not the incident commander for the scenario
- A scribe capturing timeline, decisions, assumptions and open questions
- Responders from engineering, support, communications and leadership as appropriate
- A scenario grounded in a real risk from the service catalog or threat model
- A no-blame ground rule agreed by every participant before the first inject
Steps
1. Define scope and objectives
Choose one scenario, write three measurable objectives and state what is explicitly out of scope so the discussion does not drift.
Expected state: Every participant can restate the objectives and the scenario in one sentence.
2. Build the inject timeline
Write a sequence of timed events, customer reports and internal escalations that escalate from ambiguity to a confirmed major incident.
Expected state: The timeline covers detection, diagnosis, mitigation, communication and recovery within ninety minutes.
3. Brief participants and ground rules
Explain the no-blame rule, the difference between the exercise and a real incident, and how a facilitator pause or stop signal works.
Expected state: Participants agree that mistakes are discussion material and know how to stop the exercise.
4. Facilitate the exercise
Release injects on schedule, ask what the team would do next rather than what they should have done, and note decisions, owners and information gaps.
Expected state: The team reaches a mitigation decision and produces a customer communication within the exercise window.
5. Run the hot wash
Immediately after the exercise, ask each participant what worked, what failed and what they would change, while the experience is fresh.
Expected state: A prioritized observation list exists and no observation is attributed to an individual.
6. Publish actions and retest
Convert observations into actions with an owner, a due date and a verification method, then schedule a follow-up exercise for the highest-risk gap.
Expected state: Every action has an owner and a date, and the retest is on the calendar.
Decision points
Should the exercise be discussion-based or a live simulation?
- Discussion-based tabletop
- Live simulation with real systems
Start with a tabletop to expose decision gaps cheaply; use a live simulation only after the process and communication paths are stable.
Should executives participate?
- Include leadership in the scenario
- Run a technical-only session
Include leadership when the scenario requires customer or regulatory communication; decision latency at the top is often the largest gap.
Should the scenario be a past incident?
- Replay a real incident with changed details
- Use a new plausible scenario
Replay a real incident when the team is new, but change names and dates so participants reason from the injects rather than memory.
Expected outcome
The response team has practiced a realistic incident, produced a no-blame findings report and committed to dated improvements with named owners.
Verification
Review the findings report for owned and dated actions, confirm participants agree with the timeline, and check that the follow-up exercise is scheduled.
Cleanup
Archive the inject timeline and findings, delete any recording made without consent, and follow up on overdue actions at the next operations review.
Practice the decisions, not the tooling
An incident is a sequence of decisions under incomplete information. A tabletop compresses that sequence into ninety minutes and lets the team practice escalation, customer communication and trade-offs without touching production. The scenario should come from a real risk: a primary database failure, a credential leak, a regional outage or a payment provider disappearing. Write it as observable symptoms, not as a root cause the team already knows.
Build an inject timeline
Injects create pressure and ambiguity. Start with a vague customer report, add contradictory monitoring, then confirm the scope. The example below fits a ninety-minute session and leaves time for the hot wash.
09:00 Inject 1 Support reports intermittent 502s from the checkout page
09:10 Inject 2 Database CPU alert fires, but replication lag is normal
09:20 Inject 3 A second region reports healthy, traffic is not shifting
09:30 Inject 4 Payment provider status page announces a partial outage
09:40 Inject 5 A journalist asks for comment on social media
09:50 Decision Publish status page update and shift traffic
10:00 Inject 6 Failover succeeds but queue backlog grows
10:15 Hot wash What worked, what failed, what changesSafety checks and ground rules
Agree on no-blame before the first inject. The facilitator should ask “what would you do next” and record the answer without correcting it; the scribe captures the clock time, decision, owner and missing information. Do not use a real ongoing incident as material, and do not blindside a team by recording without consent. If the exercise uncovers a live production risk, stop the exercise and switch to the real incident process; that is the one case where the tabletop is abandoned.
Rollback and follow-up
There are no irreversible technical steps in a tabletop, and no production change should be made during one. The irreversible risk is social: if findings are used to blame individuals, the next exercise will produce guarded answers. Publish the report as observations and system gaps, assign actions with owners and dates, and track them like any other operational work. Schedule the retest before the findings lose urgency, ideally within one quarter.