Roll Back a Bad Deployment with kubectl rollout undo

Inspect revision history, undo to the last good revision by number, and verify the rollback actually landed by comparing live pod specs.

~15 mindeploykubernetesrollback

Task

Identify the last good revision, roll back deterministically, and verify the live state matches it rather than assuming the command worked.

When to use this

A Friday deploy spikes error rates. The team runs rollout undo with no arguments, gets an unknown revision, and is unsure whether production now runs 1.4.1 or still 1.4.2.

Prerequisites

kubectl access to the deployment
A deployment with revisionHistoryLimit not set to zero
A monitoring signal (error rate, latency) to confirm recovery

Steps

1. List the revision history before touching anything.

kubectl rollout history deployment/web -n prod

Expected output: Revisions print with change-cause annotations, showing which number corresponds to the last known-good version.

2. Inspect the candidate revision to confirm it is what you think.

kubectl rollout history deployment/web -n prod --revision=11 | grep -E "image:|Image:"

Expected output: The container image of revision 11 prints, for example registry/web:1.4.1, proving this is the revision you want before rolling back.

3. Undo to the explicit revision number, not just "previous".

kubectl rollout undo deployment/web -n prod --to-revision=11 && kubectl rollout status deployment/web -n prod --timeout=180s

Expected output: The rollout completes and the status command exits zero, meaning all pods are updated and ready on the old revision.

4. Verify the live pods actually run the old image.

kubectl get pods -n prod -l app=web -o jsonpath="{range .items[*]}{.spec.containers[0].image}{"\n"}{end}" | sort | uniq -c

Expected output: Exactly one image prints, the old 1.4.1, confirming no mixed-version pods remain during the rollback window.

5. Confirm with the monitoring signal that motivated the rollback.

curl -s "https://monitor.example.com/api/query?query=sum(rate(errors_total{service=\"web\"}[2m]))" | jq -r '.data.result[0].value[1]'

Expected output: The error rate returns to its pre-incident baseline, closing the loop from detection to verified recovery.

Troubleshooting

Problem

History shows only one revision

Solution

revisionHistoryLimit was set to 1 or 0; the old ReplicaSet is gone, so redeploy the known-good manifest from Git instead of relying on cluster history.

Problem

Rollback stuck with pods pending

Solution

Resource quota or node pressure blocks the old ReplicaSet from scaling; check describe pod events and free headroom before assuming the revision itself is broken.

Problem

Undo succeeded but errors continue

Solution

The regression was not the deployment but a config or schema change shipped with it; roll those back too, or run the forward fix, since undo only reverts the pod template.

Limitations

rollout undo reverts the pod template only; ConfigMaps, secrets, and database schemas are outside its reach and must be reverted separately.
Git is the real source of truth; treat cluster revision history as a convenience and reconcile the repo after any rollback.

Undo by number, then prove it landed

Bare rollout undo rolls to “the previous revision”, which during a chaotic incident may not be the one you mean. Listing history, undoing to an explicit revision number, and then verifying the live image closes every gap between intention and reality.

kubectl rollout history deployment/web -n prod
# REVISION  CHANGE-CAUSE
# 10        deploy 1.4.1
# 11        deploy 1.4.2   <-- bad

kubectl rollout undo deployment/web -n prod --to-revision=10
kubectl rollout status deployment/web -n prod --timeout=180s

# Prove what production runs now
kubectl get pods -n prod -l app=web \
  -o jsonpath="{.items[*].spec.containers[0].image}"
# registry/web:1.4.1

The verification is two-layered

A zero exit code from rollout status proves the controller finished; the image query proves the running pods are on 1.4.1. Both are needed, because a completed rollout with a wrong revision is silent, and a stalled rollout can look complete while old pods linger.

Reconcile Git immediately

The cluster now runs a revision that Git HEAD does not describe. Commit the revert or re-pin the manifest right after verification, or the next routine deploy re-ships the bad version over your rollback, which is how one incident becomes two.