阅读中文版本

Post-Rollout Verification Workflow

Verify key user journeys, error rates, and latency after a rollout and roll back quickly if any threshold is breached.

Overview

Confirm a rollout is healthy before declaring it complete and enable fast rollback.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• Dashboards for errors and latency
• Synthetic journey checks
• A rollback mechanism

Procedure

1. Check error and latency dashboards

Compare error rate and latency against the pre-rollout baseline for the same traffic.

Expected state: Metrics sit within the agreed SLO thresholds.

2. Run synthetic critical journeys

Execute login, checkout, and search journeys against the new version.

Expected state: Every critical journey passes end to end.

3. Watch for a soak period

Observe metrics for the defined soak window before increasing traffic.

Expected state: No regressions appear during the soak window.

4. Declare success or roll back

If thresholds hold, declare success; otherwise trigger rollback and stop the rollout.

Expected state: The release is either complete or safely reverted.

curl -fsS https://app.example.com/health
npx playwright test smoke --reporter=line || echo 'rollback required'

Verification

Confirm metrics match baseline, synthetic journeys pass, and the soak window completed without regressions.

Decision points

Automate rollback or keep it manual?

• Automatic on threshold breach
• Manual decision

Recommendation: Automate rollback for clear thresholds; keep manual for ambiguous signals.

How long to soak?

• One full traffic cycle
• A few minutes

Recommendation: Soak for at least one representative traffic cycle to catch periodic issues.

Common failures

• Declaring success without checking metrics.
• Soak too short to catch periodic regressions.
• No rollback path wastes the verification window.