Automate Canary Analysis and Rollback with Argo Rollouts

Replace manual canary judgment with an AnalysisTemplate that queries error rate and latency metrics and promotes or rolls back automatically.

~45 mindeploykubernetesobservability

Task

Configure an automated canary that compares metrics between canary and stable and rolls back without human reaction time.

When to use this

Canary deploys depend on whoever is awake watching a dashboard. A bad release sat at five percent traffic for forty minutes on a Saturday because nobody was looking.

Prerequisites

A Kubernetes cluster with Argo Rollouts installed
A metrics source such as Prometheus exposing per-version error rate and latency
A service mesh or rollout traffic router to split traffic

Steps

1. Define the analysis queries against your metrics.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      successCondition: result[0] < 0.02
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{version="canary",code=~"5.."}[2m]))
            / sum(rate(http_requests_total{version="canary"}[2m]))

Expected output: The template fails the canary when error rate exceeds two percent on two consecutive one-minute evaluations.

2. Attach the analysis to a phased rollout.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: success-rate }] }
        - setWeight: 25
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: success-rate }] }
        - setWeight: 50
        - setWeight: 100

Expected output: Traffic moves 5, 25, 50, 100 percent with an automated gate between each phase.

3. Verify the rollback path with a deliberately bad release.

kubectl argo rollouts set image rollout/app app=app:broken && kubectl argo rollouts get rollout app --watch

Expected output: The rollout aborts at the first failing analysis, traffic returns to the stable version, and the watch shows the rollback completing automatically.

4. Add latency as a second guardrail.

metrics:
  - name: p99-latency
    interval: 1m
    successCondition: result[0] < 800
    failureLimit: 3
    provider:
      prometheus:
        query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{version="canary"}[2m])) by (le))

Expected output: A release that is functionally correct but twice as slow also fails the gate and rolls back.

5. Confirm alerting on aborted rollouts.

kubectl argo rollouts list rollouts && kubectl get events --field-selector reason=RolloutAborted | tail -3

Expected output: Aborted rollouts emit events, so the on-call learns the deploy failed and rolled back even while asleep.

Troubleshooting

Problem

Analysis never gets data

Solution

The query returns empty because the version label differs; align the label in the query with what the rollout injects, and test the query in Prometheus first.

Problem

Flapping between fail and pass

Solution

Failure counts accumulate across evaluations; set a count-based successCondition window and tune failureLimit so transient blips do not abort healthy releases.

Problem

Prometheus is down during deploy

Solution

The provider errors count as inconclusive; set errorLimit explicitly so infrastructure failures block promotion rather than silently passing it.

Limitations

Automated analysis is only as good as the metrics; a service without per-version labels cannot be compared.
Statistical gates need meaningful traffic at each step; five percent of low traffic may generate too few requests to judge.

Judgment that does not blink

A human watching a dashboard has a reaction time measured in minutes and an attention span measured in hours; a bad canary needs a verdict in seconds. Argo Rollouts moves the verdict into the cluster: an AnalysisTemplate queries per-version error rate and latency, each traffic increase is gated on the result, and two failing evaluations abort the rollout and return traffic to stable before anyone wakes up.

metrics:
  - name: error-rate
    interval: 1m
    successCondition: result[0] < 0.02
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus:9090
        query: |
          sum(rate(http_requests_total{version="canary",code=~"5.."}[2m]))
          / sum(rate(http_requests_total{version="canary"}[2m]))

Promote slowly, judge quickly

The step schedule and the analysis cadence serve different purposes: weights of 5, 25, 50 limit blast radius, while one-minute evaluations with failureLimit two remove bad releases within about three minutes of symptoms. Slow promotion plus fast judgment is the combination that protects users and does not bore the deploy pipeline.

Test the rollback before you need it

An automated rollback you have never exercised is a hypothesis. Deploy a deliberately broken image, watch the analysis abort the rollout, and confirm traffic returns to stable. That rehearsal belongs in staging on a schedule, because the rollback path is the one piece of the system you can never afford to discover is broken during a real incident.