阅读中文版本

Mitigation-First Incident Workflow

Stop the bleeding before diagnosing, using reversible mitigations, so users recover while the investigation continues.

Overview

Restore service quickly with reversible mitigations before completing root-cause analysis.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• A rollback or flag mechanism
• Traffic controls
• A communication channel

Procedure

1. Stabilise the most affected surface

Apply the fastest reversible mitigation, such as disabling a flag or rolling back.

Expected state: User-facing errors start to fall.

2. Confirm the user impact is reducing

Watch the key metric recover rather than assuming the mitigation worked.

Expected state: The impact metric trends back to normal.

3. Communicate the mitigation

Tell stakeholders what was changed and the current state, without claiming resolution.

Expected state: Stakeholders know the mitigation is in place.

4. Continue diagnosis separately

Investigate the root cause in parallel without blocking the mitigation.

Expected state: Mitigation and investigation proceed independently.

# fastest reversible mitigation: turn off the suspect flag
flagctl disable new_checkout
echo 'mitigation applied; watching error rate'

Verification

Confirm the impact metric recovered after the mitigation and the change is documented and reversible.

Decision points

Mitigate or investigate first?

• Mitigate first
• Diagnose first

Recommendation: Mitigate first to stop user harm; diagnosis can continue alongside.

Choose the fastest or most complete fix?

• Fastest reversible mitigation
• Complete fix

Recommendation: Prefer the fastest reversible mitigation during the incident.

Common failures

• Long diagnosis delays user relief.
• Irreversible mitigations remove escape routes.
• Claiming resolution before the metric recovers misleads stakeholders.