Roll Out Config Changes Without Restarting Everything

Version ConfigMaps, roll pods gradually with maxUnavailable 0, and keep serving when a configuration key is missing.

~30 minkubernetesconfigdeployment

Task

Apply configuration changes during business hours without dropping requests or restarting every pod at once.

When to use this

A ConfigMap edit updates a timeout, but the deployment only reads it at startup, so three of nine pods silently keep the old value for days.

Prerequisites

Kubernetes 1.27 or newer
Helm 3 for the checksum annotation, or an equivalent template tool
A Deployment with at least three replicas and a readiness probe

Steps

1. Add a checksum annotation so a config change produces a new ReplicaSet.

spec:
  template:
    metadata:
      annotations:
        checksum/config: "{{ include (print $.Template.BasePath \"/configmap.yaml\") . | sha256sum }}"

Expected output: Editing the ConfigMap changes the pod template hash, so a rolling update starts.

2. Keep full capacity during the rollout.

strategy:
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Expected output: A new pod becomes ready before an old pod terminates, so available replicas never drop.

3. Watch the rollout and confirm old pods disappear one at a time.

kubectl rollout status deployment/payments -n prod --timeout=300s && kubectl get pods -n prod -l app=payments -o wide

Expected output: Pods are replaced one by one, and every old pod terminates only after a replacement is ready.

4. Validate configuration at startup so a malformed value never serves traffic.

const timeout = Number(process.env.REQUEST_TIMEOUT_MS ?? 3000);
if (!Number.isFinite(timeout) || timeout <= 0) {
  throw new Error('invalid REQUEST_TIMEOUT_MS');
}

Expected output: A malformed value fails the readiness probe instead of serving with a broken timeout.

Troubleshooting

Problem

The rollout stalls with one pod pending

Solution

With maxUnavailable 0 a pending pod blocks progress. Check readiness logs and resource limits, and fix capacity before retrying.

Problem

Editing the ConfigMap changes nothing

Solution

Environment variables are captured at container start; only volume mounts update in place, and the application must watch the file.

Limitations

Volume-mounted ConfigMap updates propagate with a delay and are not atomic across keys.
Zero-downtime config changes assume the old and the new configuration are both valid at the same time.

Config is part of the release

A ConfigMap edit looks harmless, but it changes the behavior of every process that reads it. Deployment pods do not automatically restart, partially because Kubernetes cannot know whether the new configuration is safe to roll out gradually and partly because env-var configuration is read only once at container start. The result is a fleet where some pods use the new value and some use the old one, with no rollout history explaining the difference. Treat configuration as a release artifact.

Trigger a controlled rollout

The checksum annotation turns a ConfigMap change into a pod template change, which gives you a normal rolling update with history and rollback. Pair it with maxUnavailable: 0 and a surge of one so capacity never drops, and make sure the readiness probe reflects real serving ability rather than process liveness. If the service cannot tolerate even one extra pod, fix that constraint before claiming zero downtime, because a rolling update needs somewhere to put the new pod.

strategy:
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Read config defensively

A configuration rollout means old and new values coexist for a few minutes, so both must be valid. Validate at startup and fail readiness when a value is missing or malformed; a pod that cannot serve correctly should not receive traffic. For values that change frequently, prefer a watched volume mount or a feature flag service, and keep the restart path for structural changes. Log the effective configuration with secrets redacted, so an incident review can tell which value a pod actually used.