Run Post-Deploy Smoke Checks with Automatic Rollback

Verify the new version with real requests, retry briefly, and roll back automatically when the checks keep failing.

~30 mindeploymentsmoke-testrollback

Task

Catch a broken deploy within a minute and return to the last good version without waiting for a human.

When to use this

A deploy starts fine but every request returns 500 because an environment variable was renamed. Health checks pass and the error appears only in the application log.

Prerequisites

A deploy pipeline that can run steps after the rollout
kubectl access to the target namespace
A health endpoint that exercises a real dependency

Steps

1. Write a smoke check that exercises the API and its database dependency.

#!/usr/bin/env bash
set -euo pipefail
for i in $(seq 1 10); do
  code=$(curl -s -o /tmp/body -w '%{http_code}' https://app.example.com/api/health)
  if [ "$code" = "200" ] && grep -q '"db":"ok"' /tmp/body; then
    exit 0
  fi
  sleep 6
done
echo "smoke check failed" >&2
exit 1

Expected output: The check passes only when the API and its database dependency both respond correctly.

2. Run the check immediately after the rollout in the same pipeline.

kubectl rollout status deployment/payments -n prod --timeout=180s && ./scripts/smoke.sh

Expected output: A broken release fails the pipeline within about a minute, before most users encounter it.

3. Roll back automatically when the smoke check fails.

- name: Roll back on failed smoke test
  if: failure()
  run: |
    kubectl rollout undo deployment/payments -n prod
    kubectl rollout status deployment/payments -n prod --timeout=180s

Expected output: The previous ReplicaSet is restored and ready, and the pipeline records the rollback reason.

4. Re-run the smoke check after the rollback to confirm recovery.

./scripts/smoke.sh

Expected output: Checks pass on the restored version, proving the failure came from the release rather than the environment.

Troubleshooting

Problem

Smoke checks fail during a normal deploy

Solution

The check may race the ingress or a cold cache. Add a short warm-up and retry loop, but keep the total budget under two minutes.

Problem

Rollback fails as well

Solution

The problem is likely data or configuration shared by both versions. Stop the pipeline and escalate instead of looping rollbacks.

Limitations

Smoke checks cover a few paths, not full regression, and cannot replace canary analysis.
Automatic rollback cannot undo migrations or side effects already written.

Readiness is not correctness

A readiness probe answers one question: should this pod receive traffic? It does not prove that the application can serve the request path a user cares about. A renamed environment variable, a wrong database URL, or a missing migration can leave a pod perfectly ready and completely broken. Post-deploy checks close that gap by making real requests through the real edge, and by accepting a 200 only when the body shows the dependency is healthy too.

Retry, then fail, then roll back

Deploys are noisy: a new pod may need a second to warm a connection pool, and the ingress may take a moment to notice endpoints. Retry the check a handful of times with a short sleep, but bound the total time so a genuinely broken release fails quickly. On failure, roll back in the same pipeline run, because that is where the context and the previous revision are known. Record the rollback and the failing response in the run summary so the next investigation starts with evidence.

for i in $(seq 1 10); do
  code=$(curl -s -o /tmp/body -w '%{http_code}' https://app.example.com/api/health)
  [ "$code" = "200" ] && grep -q '"db":"ok"' /tmp/body && exit 0
  sleep 6
done
exit 1

Choose paths that matter

Smoke checks should cover the entry points users reach first: the homepage, the login flow, the primary API call, and any dependency that has failed before. Keep the suite small enough to run in under a minute, and version it with the application so it evolves with the API. Do not put destructive operations in a smoke test, and never let it write production data that cannot be cleaned up. The goal is fast, unambiguous evidence, not coverage.