阅读中文版本

Flaky Test Quarantine and Escalation Workflow

Quarantine flaky tests out of the blocking path while tracking them to a fix with an owner and deadline.

Overview

Keep CI trustworthy by isolating flaky tests without forgetting them.

This workflow is reusable across teams and environments; adapt the commands to your tooling while keeping the sequence and checkpoints intact.

Environment and prerequisites

• A CI pipeline
• A flaky-test tracker
• Test rerun capability

Procedure

1. Detect flakiness

Identify tests that pass and fail non-deterministically across reruns.

Expected state: A list of flaky tests exists with evidence.

2. Quarantine from the gate

Move flaky tests out of the blocking path but keep running them.

Expected state: The main gate stops being blocked by flakes.

3. Assign owner and deadline

Give each quarantined test an owner and a fix deadline.

Expected state: Quarantined tests are not forgotten.

4. Fix or delete

Fix the root cause or delete the test if it no longer has value.

Expected state: The quarantine list shrinks to zero.

# rerun to detect flakiness
npx vitest run --repeat=5 --reporter=json > reruns.json

Verification

Confirm the quarantine list has owners and deadlines and trends toward empty.

Decision points

Delete or fix flaky tests?

• Fix if it guards real behaviour
• Delete immediately

Recommendation: Fix tests that guard important behaviour; delete those that do not.

Quarantine silently?

• Quarantine with owner and deadline
• Just disable

Recommendation: Always attach an owner and deadline so quarantine is temporary.

Common failures

• Ignoring flaky tests erodes trust in CI.
• Silent disabling means they never return.
• Deleting all flakes removes valuable coverage.