Fail the Deploy When a Rollout Stalls

Gate the pipeline on kubectl rollout status with a timeout, inspect the stuck ReplicaSet, and roll back instead of hanging.

~25 minkubernetesdeploymentpipeline

Task

Turn a stalled rolling update into a failed, reversible pipeline step instead of a silent partial deploy.

When to use this

A new pod cannot pull its image because the tag does not exist. The pipeline reports success after applying manifests, and half the fleet keeps running the old version.

Prerequisites

kubectl 1.27 or newer in CI
A Deployment with a readiness probe
Permission to read events and ReplicaSets

Steps

1. Apply the manifests and wait for the rollout inside one pipeline step.

kubectl apply -f k8s/ -n prod && kubectl rollout status deployment/payments -n prod --timeout=240s

Expected output: The step blocks until every pod is updated and ready, or fails after four minutes.

2. Diagnose a stalled rollout before deciding to roll back.

kubectl describe deployment/payments -n prod | sed -n '/Conditions/,$p' && kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expected output: Conditions show Progressing=False with a reason such as ImagePullBackOff or an unavailable replica count.

3. Roll back and wait for the previous revision to be ready.

kubectl rollout undo deployment/payments -n prod && kubectl rollout status deployment/payments -n prod --timeout=240s

Expected output: The previous revision becomes ready and the pipeline records the failed revision.

4. Check that the image reference exists before touching the cluster.

docker manifest inspect ghcr.io/acme/payments:1.5.0 > /dev/null

Expected output: The pipeline fails early when the tag is missing or the registry denies access.

Troubleshooting

Problem

rollout status times out but pods are running

Solution

A readiness probe never passes. Check the probe path, port, and startup time before raising the timeout.

Problem

The deployment reports complete while old pods remain

Solution

maxSurge and maxUnavailable allow overlap. Check ReplicaSet counts and the deployment's updatedReplicas field.

Limitations

rollout status waits for the Deployment, not for traffic to be correct; pair it with smoke checks.
A long timeout slows every failing deploy, so keep it close to real startup time.

Applying is not deploying

kubectl apply returns as soon as the API server accepts the object. It does not wait for pods to be created, scheduled, or ready. A pipeline that treats apply as success can finish green while every new pod is stuck pulling an image, leaving the service on the old version with no record of the failure. kubectl rollout status blocks until the Deployment reports completion, and it exits non-zero when the timeout expires, which is exactly the signal a pipeline needs.

Put the wait in the same step

Run apply and rollout status in one step so a failure is attributed to the deploy rather than to a later stage. Set the timeout from measured startup time rather than a round number; too long hides a broken release, too short fails healthy ones. When the wait fails, gather the Deployment conditions and recent events before rolling back, because the reason is usually unambiguous: an image pull error, a failing readiness probe, or insufficient resources.

kubectl apply -f k8s/ -n prod
kubectl rollout status deployment/payments -n prod --timeout=240s

Roll back with evidence

A failed rollout should be undone in the same run. kubectl rollout undo restores the previous ReplicaSet, and a second rollout status confirms that the rollback itself completed. Record both the failed revision and the rollback in the run summary. Then move the failure earlier: check that the image tag exists with docker manifest inspect before applying, so the most common cause never reaches the cluster.