中文
devexamples
← Workflows

阅读中文版本

Zero-Downtime Rolling Deployment

Roll out a new version in batches while readiness checks drain old instances, monitor SLOs, and roll back automatically when error or latency budgets are breached.

intermediate~45 mindeploymentkubernetesreliability

Objective

Replace every running instance with a new version without dropping in-flight requests or routing traffic to unready processes.

Environment

  • Kubernetes Deployment or a VM pool behind a load balancer
  • Readiness and liveness endpoints
  • Graceful shutdown that drains in-flight requests
  • Error-rate and latency dashboards

Steps

1. Verify probes and shutdown behavior

Confirm readiness fails during startup and shutdown, liveness is process-local, and SIGTERM drains requests before exit.

Expected state: A new instance is removed from rotation before it stops accepting connections.

2. Deploy with zero unavailable capacity

Set maxUnavailable to 0 and maxSurge to 1 so new pods become ready before old pods are removed.

Expected state: At least the desired replica count is available throughout the rollout.

3. Watch SLOs during the rollout

Monitor 5xx rate, p95 latency, and readiness failures for at least one batch before continuing.

Expected state: Metrics remain within the release budget; otherwise pause or roll back.

4. Complete and soak

Finish the rollout, confirm the version endpoint reports the new build, and observe the system for a defined soak period.

Expected state: All replicas run the new version with normal error and latency rates.

5. Clean up old resources

Remove old ReplicaSets, temporary feature flags, and migration shims only after the soak period passes.

Expected state: No obsolete version remains routable or referenced by configuration.

Decision points

Should the first batch be a canary or a full rolling update?

  • Canary with a small traffic slice
  • Standard rolling update

Use a canary when the change affects critical paths or dependencies; use rolling updates for low-risk changes with strong automated tests.

What triggers an automatic rollback?

  • Error rate above the release budget for five minutes
  • Any readiness failure
  • Manual decision only

Automate rollback on sustained SLO breach; readiness failures should pause the rollout rather than retry indefinitely.

Expected outcome

The new version serves all traffic with no dropped requests, no readiness-related 5xx spike, and an auditable rollout that can be paused or reversed.

Verification

Compare error rate and p95 latency before, during, and after the rollout; confirm the version endpoint on every replica; send sustained traffic during the deployment and assert no connection resets.

Cleanup

Delete old ReplicaSets and temporary rollout flags; restore any temporarily increased timeouts or replica counts.

Environment and preconditions

A rolling deployment is only zero-downtime if the platform removes instances from rotation before they stop serving, and if the application drains in-flight work. Verify both before changing the deployment strategy.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
readinessProbe:
  httpGet:
    path: /readyz
    port: 3000
  periodSeconds: 5

Steps

  1. Deploy the new image and watch kubectl rollout status deployment/app.
  2. Observe error rate, p95 latency, and readiness for the first batch.
  3. Pause or roll back if the release budget is breached.
  4. Continue batches and soak the new version.
  5. Remove old ReplicaSets and temporary flags after the soak.

Verification

The version endpoint should report the new build on every replica. A sustained load test during the rollout must show no connection resets and no 5xx spike attributable to readiness. Compare dashboards with the pre-deploy baseline.

Rollback

kubectl rollout undo deployment/app restores the previous ReplicaSet. Rollback must also reverse configuration and schema changes that are not backward compatible; that is why expand/contract migrations are required.