Gate a Canary on Error Rate and Latency

Compare canary metrics with a Prometheus query, abort automatically on regression, and fail closed when the canary has no traffic.

~35 minkubernetescanarymetrics

Task

Let real traffic decide whether a release continues, instead of relying on a fixed wait time and a human guess.

When to use this

A canary passes health checks but raises p95 latency by 40 percent on one endpoint; nobody watches the dashboard for twenty minutes and the rollout completes.

Prerequisites

Prometheus scraping canary and stable pods with a shared label
Argo Rollouts installed with the kubectl plugin
A defined success threshold agreed with the service owner

Steps

1. Define an analysis that fails when the canary error ratio exceeds one percent.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{job="payments",track="canary",status=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{job="payments",track="canary"}[2m]))

Expected output: The analysis fails when the canary 5xx ratio exceeds one percent in any measurement.

2. Reference the analysis from the rollout strategy at a low traffic weight.

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - analysis:
          templates:
            - templateName: canary-error-rate

Expected output: The rollout pauses at ten percent and proceeds only after five successful measurements.

3. Watch the abort path when a release is deliberately broken.

kubectl argo rollouts get rollout payments -n prod --watch

Expected output: Status becomes Degraded, traffic returns to stable, and the analysis run is recorded as failed.

4. Fail closed when the canary receives no traffic at all.

- name: traffic-present
  interval: 1m
  count: 2
  successCondition: result[0] > 0
  provider:
    prometheus:
      address: http://prometheus.monitoring:9090
      query: |
        sum(rate(http_requests_total{job="payments",track="canary"}[2m]))

Expected output: A canary with zero requests fails the analysis instead of passing with no evidence.

Troubleshooting

Problem

The analysis fails immediately with no data

Solution

The query selector does not match canary pods. Check the track label and confirm the canary Service actually receives traffic.

Problem

The analysis passes on a tiny sample

Solution

Low traffic makes ratios noisy; require a minimum request rate or extend the analysis before promoting.

Limitations

Metrics gates cannot catch correctness bugs that do not move error rate or latency.
Scrape interval and Prometheus retention limit how quickly a regression can be detected.

A canary without a gate is a slow deploy

Sending ten percent of traffic to a new version only helps if something decides, from data, whether the version is good. Otherwise the rollout waits an arbitrary number of minutes and promotes, which is a rolling update with extra steps. Argo Rollouts analysis templates run a query on an interval, compare the result with a success condition, and abort the rollout when the condition fails. The value comes from choosing a metric that moves before users complain, and from failing closed when the metric cannot be evaluated.

Query the canary, not the service

The query must select only canary pods. If it aggregates over the whole service, a canary error spike is diluted by stable traffic and the gate passes. Label both tracks consistently and keep the label in the metric, so the same query can compare canary against stable later. Set failureLimit deliberately: a single bad measurement can be noise, but two consecutive failures usually mean a real regression.

successCondition: result[0] < 0.01
provider:
  prometheus:
    address: http://prometheus.monitoring:9090
    query: |
      sum(rate(http_requests_total{job="payments",track="canary",status=~"5.."}[2m]))
      /
      sum(rate(http_requests_total{job="payments",track="canary"}[2m]))

Treat missing evidence as failure

An empty Prometheus result can be interpreted as “no errors”, which is exactly the wrong conclusion when the real cause is a broken scrape or a canary with no traffic. Add a companion metric that requires a non-zero request rate, and make the abort path return all traffic to stable. Record the analysis run with the release, because “the canary passed” is only meaningful with the query and the numbers that backed it.