Rehearse a Rollback Before You Need It

Time a real rollback in staging, verify the old version still handles new data, and write the exact commands into the release runbook.

~30 minreleaserollbackdrill

Task

Prove that the previous version still works after a release, and that the rollback takes minutes rather than hours.

When to use this

A release is rolled back for the first time during an incident. The old image no longer starts because a migration changed a column, and the rollback takes ninety minutes.

Prerequisites

A staging environment that mirrors production topology
At least two released image versions in the registry
A release runbook that can be updated after the drill

Steps

1. Record the current revision before deploying the candidate.

kubectl rollout history deployment/payments -n staging

Expected output: The current revision number is recorded so the rollback target is unambiguous.

2. Deploy the candidate and time a rollback to the recorded revision.

kubectl set image deployment/payments api=ghcr.io/acme/payments:1.5.0 -n staging && time kubectl rollout undo deployment/payments -n staging --to-revision=41 && kubectl rollout status deployment/payments -n staging --timeout=180s

Expected output: The rollback completes inside the target window, for example under three minutes, and pods become ready.

3. Verify that the old version reads data written by the new one.

curl -fsS https://staging.example.com/api/orders/12345 | jq -r '.status'

Expected output: The request returns a valid status for a record created during the canary, proving backward compatibility.

4. Write the verified command, duration, and decision owner into the runbook.

Expected output: The runbook lists the exact rollback command, expected duration, and who can call a rollback without waiting for approval.

Troubleshooting

Problem

Rollback fails because the previous image was deleted

Solution

Keep at least two previous image digests with a retention policy longer than the release window.

Problem

Rollback succeeds but errors continue

Solution

The failure is in data or a dependency, not the code. Roll forward or restore, and record the finding for the next drill.

Limitations

A staging drill cannot reproduce production data volume or third-party state.
Rolling back code does not undo migrations; only expand and contract designs make it safe.

The first rollback should not be in production

Most teams discover that rollback is broken at the worst possible moment. The image tag was overwritten, the previous revision depended on a column that a migration dropped, or nobody remembers the flag that has to be flipped with the code. A rollback drill moves those discoveries into a quiet afternoon in staging, where a failure costs an hour instead of an incident. The output of the drill is not a green check mark; it is a runbook with commands that were actually executed and timings that were actually measured.

Roll back the way an incident would

Do the drill under realistic pressure: deploy a candidate, then roll it back using only the runbook, without improvising. Time the whole operation from decision to healthy traffic, not just the kubectl call. Include the data check, because a version that starts but cannot read newly written rows has not really rolled back. If the service writes to a queue or an external system, include the effect of the old version writing to a schema the new version created.

kubectl rollout undo deployment/payments -n staging --to-revision=41
kubectl rollout status deployment/payments -n staging --timeout=180s
curl -fsS https://staging.example.com/api/orders/12345 | jq -r '.status'

Turn findings into requirements

Every drill should produce at least one concrete change: a retention policy for image digests, a migration split into expand and contract releases, a flag documented in the runbook, or a rollback owner added to the on-call rotation. Schedule the drill before major releases rather than annually, and repeat it after any change to the deploy mechanism. A rollback that has never been exercised is a plan, not a capability.