阅读中文版本

Incident Rollback Decision Workflow

Decide quickly whether to roll back or roll forward during an incident using data compatibility and blast radius.

Overview

Make a timely, evidence-based rollback or roll-forward decision during an incident.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• Deployment history
• A rollback capability
• Schema compatibility information

Procedure

1. Check whether the change is reversible

Determine if a rollback would break data or schema compatibility.

Expected state: Reversibility is known before deciding.

2. Estimate rollback cost and time

Assess how long a rollback takes and what it disrupts.

Expected state: The time cost of rollback is clear.

3. Compare against roll-forward options

Evaluate whether a forward fix is faster and safer than reverting.

Expected state: Both options have concrete estimates.

4. Decide and document

Choose one path, state the trigger to reconsider, and record the rationale.

Expected state: The decision and its review trigger are documented.

# inspect recent deploys to choose a safe revert target
kubectl rollout history deploy/app

Verification

Confirm the chosen path was executed, data stayed consistent, and the decision rationale is recorded.

Decision points

Roll back or roll forward?

• Roll back when reversible
• Roll forward when schema is one-way

Recommendation: Roll back when schema and data are compatible; roll forward when a migration cannot be undone.

Who decides?

• Incident commander
• Whoever deployed

Recommendation: The incident commander decides with engineering input to keep it fast and accountable.

Common failures

• Rolling back past a one-way migration corrupts data.
• Slow decisions extend user impact.
• No recorded rationale means the decision is re-litigated.