Tune Kubernetes Rolling Updates for Zero Downtime

Set maxSurge, maxUnavailable, readiness gates, and preStop sleep so pods replace each other without serving a single failed request.

~35 mindeploykubernetesoperations

Task

Configure a Deployment rolling update that replaces pods with zero failed requests under continuous load.

When to use this

Every kubectl apply causes a window of connection-refused errors. The team blames Kubernetes, but the deployment spec has defaults that trade availability for speed.

Prerequisites

A Kubernetes cluster with a service and deployment you can update
A load generator able to sustain traffic through the update
Permission to patch the deployment strategy

Steps

1. Record the failure baseline with default settings.

hey -n 20000 -c 20 -m GET https://app.example.com/ & kubectl apply -f deploy.yaml; wait 2>/dev/null || true

Expected output: The baseline shows a handful of connection errors during the update window, proving the default strategy drops requests.

2. Set the strategy to surge, never unavailable.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Expected output: A new pod starts before any old pod stops, so capacity never dips below the running count.

3. Gate traffic on readiness, not liveness.

readinessProbe:
  httpGet: { path: /readyz, port: 3000 }
  initialDelaySeconds: 2
  periodSeconds: 2
  failureThreshold: 3

Expected output: The service stops routing to a pod the moment readiness fails, and new pods receive traffic only after the probe passes.

4. Add a preStop sleep so endpoints propagate.

lifecycle:
  preStop:
    exec: { command: ["sleep", "5"] }

Expected output: The pod keeps serving for five seconds after termination starts, covering the interval the endpoint list takes to converge.

5. Re-run the update under load and count errors.

hey -n 20000 -c 20 -m GET https://app.example.com/ & kubectl set image deploy/app app=app:2.4.1; wait

Expected output: Zero non-200 responses across the entire rolling update, with the surge pod absorbing the handover.

Troubleshooting

Problem

Errors persist despite maxUnavailable 0

Solution

The app takes longer than the probe delays to become ready; raise initialDelaySeconds or implement a readiness endpoint that reports true only when warmed up.

Problem

Surge pod cannot schedule

Solution

The cluster lacks headroom for one extra pod; set a requests-based pod budget and ensure node autoscaling or spare capacity covers the surge.

Problem

Long requests still get cut

Solution

preStop sleep covers routing convergence, not request duration; set terminationGracePeriodSeconds above your p99 request time and drain in the app itself.

Limitations

Zero-downtime rolling requires readiness probes that reflect true serviceability; a trivially-true probe voids the whole scheme.
The surge pod needs cluster headroom; clusters running at 100 percent allocation cannot roll safely.

Defaults are fast, not safe

The default rolling update allows a quarter of the pods to be unavailable, which converts every deploy into a brief capacity cut. Setting maxUnavailable to zero and surging instead means a replacement pod runs and passes readiness before its predecessor dies. Add a preStop sleep to cover endpoint propagation delay, and the update becomes mathematically incapable of refusing a request.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

containers:
  - name: app
    readinessProbe:
      httpGet: { path: /readyz, port: 3000 }
    lifecycle:
      preStop:
        exec: { command: ["sleep", "5"] }

Readiness is the traffic switch

Kubernetes routes traffic based on readiness, so the probe is the actual deploy gate. A readiness check that always returns true lets a cold, uninitialized pod receive requests and fail them. The probe should verify real serviceability: database connectivity, cache warmth, migrations applied. Only then does maxUnavailable zero mean what it says.

The five-second secret

The most common zero-downtime failure is not scheduling or probes but propagation: the API server marks a pod terminating, but every node-side proxy still routes to it for a few seconds. The preStop sleep keeps the old pod serving through exactly that window. It looks trivial, and it removes the majority of deploy-time connection errors.