← Recipes

阅读中文版本

Switch Traffic with a Blue-Green Service Cutover

Run the new version beside the old one, flip the Service selector in one step, and keep the old ReplicaSet warm for an instant rollback.

~30 minkubernetesdeploymentrelease

Task

Release a version with a one-command cutover and a rollback that takes seconds instead of a full redeploy.

When to use this

A migration-sensitive service must not run mixed versions; a rolling update leaves old and new pods serving traffic at once and corrupts cached sessions.

Prerequisites

  • Kubernetes 1.27 or newer
  • Two Deployments (blue and green) behind one Service
  • Readiness probes that reflect real serving ability

Steps

1. Deploy green alongside blue and wait until it is fully ready.

kubectl set image deployment/payments-green api=ghcr.io/acme/payments:1.5.0 -n prod && kubectl rollout status deployment/payments-green -n prod --timeout=180s

Expected output: Green reaches full readiness while blue continues to serve every request.

2. Verify green directly before any user traffic reaches it.

kubectl port-forward svc/payments-green 8081:80 -n prod & sleep 2 && curl -fsS localhost:8081/healthz

Expected output: The health endpoint returns 200 from green, and no production traffic has touched it yet.

3. Flip the Service selector to green in a single patch.

kubectl patch svc payments -n prod --type=merge \
  -p '{"spec":{"selector":{"app":"payments","track":"green"}}}'

Expected output: Endpoints switch from blue pods to green pods at the next kube-proxy sync.

4. Roll back by patching the selector back to blue.

kubectl patch svc payments -n prod --type=merge -p '{"spec":{"selector":{"app":"payments","track":"blue"}}}'

Expected output: Traffic returns to blue within seconds, and the green deployment stays available for inspection.

Troubleshooting

Problem

The cutover produces errors for some clients

Solution

Long-lived connections keep using old pods. Drain blue with a preStop sleep and connection draining, or switch gradually with weighted routing.

Problem

The Service has no endpoints after the patch

Solution

The selector must match pod labels exactly, including any labels added by the Deployment or a mutating webhook.

Limitations

  • Both versions run during the cutover, which doubles capacity cost for that period.
  • The database schema must be compatible with both versions; use expand and contract migrations.
  • In-memory session state is lost at cutover unless sessions are externalized.

Two versions, one switch

A rolling update replaces pods gradually and runs both versions side by side for a while. For stateless services that is fine; for services where mixed versions corrupt shared state, a blue-green cutover is safer because traffic moves at a single point in time. The cost is duplicate capacity and a stricter requirement: both versions must be able to read and write the same data. Blue-green does not remove the need for backward-compatible migrations, it makes that requirement visible.

Prepare green completely

Green must be fully ready before any user reaches it. Deploy it with its own label, wait for kubectl rollout status, then exercise it through a port-forward or a temporary Service. Checking the health endpoint is the minimum; run one or two real read paths if the service has them. Only after that do you change the selector, and the change should be a single patch so the cutover is one atomic operation rather than a sequence that can stop halfway.

kubectl patch svc payments -n prod --type=merge \
  -p '{"spec":{"selector":{"app":"payments","track":"green"}}}'

Keep the rollback warm

Do not delete blue after the switch. Keep it at full replica count until the new version has served real traffic for an agreed window; the rollback is then another selector patch that takes effect in seconds. Watch error rate and latency for both tracks during that window, because the fastest signal that the cutover failed is usually at the edge, not in pod status. When the window closes, scale blue down but keep the image tag available so the previous version can be restored without a rebuild.