Automate Deployment Rollback on Failed Health Check
Wire a rollback step into the deploy pipeline that triggers when the post-deploy health check fails beyond a retry threshold.
Task
Automate kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') so pipeline failures are caught before merge and never reach production.
When to use this
A pull request merges with a broken kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') step because the workflow did not gate on it, causing a failed deploy.
Prerequisites
- Node.js 22 and pnpm available
- A CI workflow file in .github/workflows
Steps
1. Configure kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') in the project configuration file with the recommended defaults.
kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}')
Expected output: The configuration is accepted and kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') runs without errors on the first invocation.
2. Run the check locally to verify kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') produces the expected result before pushing.
pnpm test
Expected output: Local run matches CI behavior; no surprises after push.
3. Add the gate to the CI pipeline so kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') blocks merge on failure.
- name: Run add deployment rollback automation
run: pnpm test
Expected output: Pull requests with failing kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}') are blocked from merging.
Troubleshooting
Problem
The check passes locally but fails in CI
Solution
Pin dependency versions and the runtime to match CI exactly; reproduce with the same Docker image or act.
Problem
Configuration is rejected with a schema error
Solution
Check the tool version matches the config schema; older versions may not support newer fields.
Limitations
- The gate only runs on events that trigger the workflow; locally pushed commits bypass it.
- Configuration drift is possible if someone edits settings outside version control.
Gate the pipeline before merge
Start by measuring the problem. Wire a rollback step into the deploy pipeline that triggers when the post-deploy health check fails beyond a retry threshold. The most common mistake is adding the gate without first confirming it catches the issue you care about. Run kubectl rollout undo deployment/api –to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk ‘{print $1}’) against the current codebase and inspect the output before wiring it into CI.
kubectl rollout undo deployment/api --to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk '{print $1}')Make failures actionable
Once the gate is in place, make failures actionable. A failing kubectl rollout undo deployment/api –to-revision=$(kubectl rollout history deployment/api | tail -2 | head -1 | awk ‘{print $1}’) should point to the exact file and line, not a generic error. Pin the tool version in the lockfile so a upstream release does not silently change behavior. Review the gate output in every pull request just like a test result.