Rollback Decision Workflow for Bad Deploys
Define rollback triggers and decision authority in advance, execute rollback as a first-class action, and learn forward-fix versus undo per signal.
Objective
Make rollback a fast, pre-authorized response to regression signals instead of an escalation debate held during the incident.
Environment
- Deployment with a tested rollback path
- Monitoring signals wired to the deployment marker
- A pre-agreed decision owner per release window
Steps
1. Define triggers before shipping
Write the rollback triggers (error rate threshold, latency breach, conversion drop) and the observation window into the release plan before deploying.
Expected state: The triggers exist in the runbook with numeric thresholds, not adjectives.
2. Name the decision owner
Assign one person per release window authority to call rollback without convening a meeting, with a deputy for their absence.
Expected state: Everyone on shift knows who decides, and that person knows they can.
3. Deploy with the marker
Ship with a deployment marker in metrics so pre and post windows can be compared automatically against the triggers.
Expected state: Dashboards split by deploy marker, and the comparison is one query.
4. Observe, then act on the trigger
During the observation window, compare against thresholds; on breach, the owner either rolls back immediately or records a bounded forward-fix decision with a deadline.
Expected state: Every breach results in an action within minutes, never a debate without a clock.
5. Verify recovery and log the decision
After rollback, confirm signals return to baseline, then record what triggered it and what the forward fix was, feeding the next release plan.
Expected state: The rollback event is logged with cause, decision maker, and recovery evidence.
Expected outcome
Rollbacks that execute in minutes under pre-authorized authority, with each decision recorded and the threshold list improving over time.
Verification
A rollback drill executes within the target time; recent regression events show decisions made within the observation window and logged with cause.
Decide the rollback rules while calm
The rollback debate held during an incident is the expensive one: signals deteriorate while people discuss. Writing triggers and naming an authority before deploying converts the incident response into executing a decision already made.
release plan v1.8.0 (excerpt):
rollback triggers (observation 30 min):
error_rate > 2% for 5 min
p99 latency > 800ms for 5 min
checkout cvr -15% rel for 10 min
decision owner: on-call release engineer
rollback path: redeploy v1.7.3 tag (rehearsed 11 min)
incident log 14:07Z: error_rate 2.4% (6 min)
14:08Z owner calls rollback (no meeting)
14:19Z v1.7.3 live, error_rate 0.3%
14:40Z forward-fix bug filed (timeout regression)Forward-fix has a deadline or it is denial
Sometimes undoing is wrong: the fix is one line, the rollback loses a schema migration. That judgment is legitimate, but only as a bounded decision: the owner commits to a forward fix within a stated time and rolls back if the deadline passes. An open-ended forward-fix is just hope with extra steps.
Rehearse the path you will depend on
An untested rollback is a hypothesis, and incidents are a bad time to test hypotheses. The rollback path (previous tag, config, migration state) belongs in release rehearsal, with its measured duration written into the plan.