Canary Release with Metric Gates
Route a small slice of traffic to a new version, compare it against the stable baseline on explicit metrics, and promote or roll back based on evidence.
Objective
Detect regressions in a new version with limited user exposure before promoting it to all traffic.
Environment
- Traffic splitting at the load balancer, service mesh, or ingress
- Per-version metrics for errors, latency, and saturation
- Automated or one-command rollback
- A defined success budget for the canary
Steps
1. Establish the baseline
Record error rate, p95 and p99 latency, throughput, and key business metrics for the stable version over a representative window.
Expected state: A baseline exists with enough traffic to distinguish signal from noise.
2. Route a small canary slice
Send 1–5% of traffic to the new version, keeping the rest on stable. Ensure requests are distributed consistently rather than by a single client.
Expected state: The canary receives real traffic while the blast radius stays small.
3. Compare against explicit gates
Evaluate the canary against the baseline on error rate, latency, saturation, and feature-specific metrics for a defined duration.
Expected state: The canary either meets every gate or is held back.
4. Promote, hold, or roll back
Increase traffic in stages when gates pass; stop routing to the canary on any hard failure; investigate ambiguous results before continuing.
Expected state: Every decision is tied to metrics and recorded.
5. Complete the rollout
Move to 100% after the final gate, keep the old version available for a rollback window, then remove it.
Expected state: All traffic runs the new version with normal metrics.
Decision points
What canary percentage should be used first?
- 1–5% with a longer observation window
- 25–50% for faster feedback
Start small for high-risk changes; increase only after the first gate passes with enough samples.
What is a hard rollback signal?
- Any increase in 5xx beyond the release budget
- A single error in logs
Use a statistically meaningful threshold tied to the SLO; a single error is usually noise, a sustained breach is not.
Expected outcome
The new version reaches production with measured evidence that it does not regress errors, latency, or core business behavior.
Verification
Compare canary and stable cohorts on the same dashboard and time window; confirm the error budget is not exhausted, latency percentiles are within target, and no new alerts fire.
Cleanup
Remove canary routing rules, old version capacity, and temporary dashboards after the rollback window closes.
Metrics before traffic
A canary is only useful if you know what “healthy” looks like. Record the stable baseline first, then define the gates: error rate, p95/p99 latency, saturation, and the business metric the change is supposed to affect.
canary:
steps:
- setWeight: 5
- pause: { duration: 10m }
- setWeight: 25
- pause: { duration: 20m }
- setWeight: 50
- pause: { duration: 20m }Run the gates
- Send 1–5% of traffic to the new version.
- Compare cohorts over the same window.
- Promote only if every gate passes.
- Roll back immediately on a hard failure.
Verification
- Canary and stable are compared on identical dashboards.
- Error and latency percentiles stay within the release budget.
- Feature metrics move in the expected direction.
- The rollback path was tested before the release.
- The old version remains available until the rollback window closes.
Pitfalls
A canary with too little traffic produces noisy comparisons. Routing all requests from one region or client biases the sample. Promoting because “no alerts fired” without looking at the baseline can miss gradual regressions. Define gates in advance and do not move them mid-release.