Flaky Test Eradication Program Workflow

Detect flakes with repeated runs, quarantine with expiry, fix root causes in priority order, and prevent reintroduction with retries-off gates.

intermediate~35 mintest-strategyflaky-testsci

Objective

Reduce the flake rate of the suite to near zero through detection, time-boxed quarantine, root-cause fixes, and gates that keep new flakes out.

Environment

A CI system with per-test retry visibility or repeat-run capability
A quarantine mechanism (tag, folder, or config)
Time budget allocated for fixing, not just skipping

Steps

1. Measure the flake rate first

Run the suite repeatedly across a window, computing per-test failure variance to rank tests by flakiness rather than by anecdote.

Expected state: A ranked flake list exists with observed failure rates per test.

2. Quarantine with expiry and owner

Move proven flakes to a quarantine lane that runs separately, each entry carrying owner, root-cause hypothesis, and an expiry that routes to escalation.

Expected state: The main suite is green and trustworthy; quarantine is visible and time-boxed.

3. Fix by blast radius, not by ease

Fix quarantined tests in order of the damage they hide: tests covering payment or auth paths before cosmetic ones, using seeds and order-shuffle to reproduce.

Expected state: High-blast-radius tests return to the main suite first, with reproducing seeds documented.

4. Turn retries off in the main lane

Ban automatic retries in the main suite so new flakes surface immediately, and treat any rerun-until-green habit as a reportable metric.

Expected state: A red main-suite run means a real failure or a new flake, never a shrug.

5. Gate on flake rate

Add a CI check comparing per-PR failure variance for touched tests, blocking merges that introduce measurable flakiness.

Expected state: A PR whose touched tests fail intermittently across repeated CI runs fails the variance check.

Expected outcome

A suite whose flake rate is measured near zero, with quarantine empty or expiring, retries banned in the main lane, and variance gates preventing new flakes.

Verification

Repeated suite runs over a week show a stable pass rate; the quarantine lane holds zero entries past expiry; retry counters in CI read zero.

Rerun-until-green is the flake factory

Every automatic retry teaches the team that red might mean nothing, and real regressions inherit that doubt. Eradication inverts the incentives: measure flakes, quarantine honestly, fix by blast radius, and make the main lane incapable of hiding them.

measurement (3 runs x 5 nights):
  flake rate: 4.7% of tests fail intermittently
  top offenders:
    payment/e2e_flow        38% fail rate  (blast: high)
    search/debounce         31%             (medium)
    ui/header_snapshot      29%             (low)

quarantine (expiry 14d, owner attached):
  fixed first: payment (seed + fake clock)
  root causes: 2 timing races, 1 order-dep state

main lane: retries DISABLED
  red run = real failure or new flake (escalates)
variance gate: touched tests must pass 3/3 runs

Fix in blast-radius order

A flaky payment test hides charge bugs; a flaky header snapshot hides a pixel. Both deserve fixing, but the payment test leaves quarantine first, because the cost of what each flake conceals differs by orders of magnitude.

Retries-off is the ratchet

Retries are compassionate in the moment and corrosive over time: they pay flake debt with trust in every future red run. Turning them off in the main lane hurts for a week and then converts every flake into a visible, actionable event instead of background noise.