Incident Command
Incident command assigns clear roles such as commander, communications lead, and operations lead during an outage so responders act instead of negotiating who decides.
Definition
Incident command is a role-based coordination structure borrowed from emergency response, giving one incident commander authority over priorities while separate leads handle communication and hands-on mitigation.
Context and scope
Activated for high-severity incidents that span teams, threaten customer-facing SLOs, or require coordinated decisions under time pressure.
Common misconceptions
- The incident commander is the most senior engineer and does the debugging
- Command means management authority and ordering people around
- Every incident needs the full role structure regardless of size
Anti-patterns
- Running a multi-team outage as an unstructured group chat
- Having the commander debug while also coordinating stakeholders
- Switching commanders without a written handoff
Related concepts
Authoritative source
How it works
When an incident is declared, a responder takes the incident commander role and assigns a communications lead and an operations lead. The commander sets priorities, approves mitigation plans, and keeps the response focused; the operations lead runs the hands-on work with subject experts; the communications lead updates stakeholders on a fixed cadence. A scribe records decisions and timestamps.
When to use it
Use incident command for high-severity events that span teams, threaten customer-facing SLOs, or require coordinated decisions. Small, single-team issues can be handled with a lightweight version. Compared with ad hoc coordination, the structure removes ambiguity about who decides, but adds roles and ceremony that would slow a trivial fix.
Trade-offs
Role assignment costs a few minutes and requires trained people. Without clear authority, responders hesitate; with too much hierarchy, experts are silenced. The commander should not debug, because context-switching destroys situational awareness. Handoffs need explicit summaries so authority does not evaporate during a long incident.
Example
A primary database fails over at 02:10. The on-call engineer declares SEV-1, becomes commander, and names an operations lead to run the failover and a communications lead to post updates every fifteen minutes. The scribe logs each action, and the incident closes at 02:52.
Misconceptions
The commander is not necessarily the best debugger or the most senior engineer. Command is not management authority or permission to bark orders; it is a decision-making service. And an incident does not need a perfect hierarchy, because roles can be combined in small responses as long as one person owns priorities.
Related concepts
A postmortem reviews the response; a runbook supplies known mitigation steps; an error budget defines when the response must escalate.