Quarantine Flaky Tests and Fix Them for Real

Detect flaky tests with repeat runs, move them to a quarantine suite that does not gate, diagnose the root cause, and graduate them back.

~35 mintestingciflakiness

Task

Identify nondeterministic tests, isolate them without losing signal, and return them to the gating suite only after a root-cause fix.

When to use this

The suite fails randomly about once in ten runs. The team reruns until green, and real regressions now hide inside the retry noise.

Prerequisites

A test suite with suspected flakiness
CI time to run the suite repeatedly for detection
A quarantine mechanism, such as a tag or separate job

Steps

1. Measure flakiness with repeat runs.

for i in $(seq 1 20); do npx vitest run --reporter=json > /tmp/run-$i.json 2>/dev/null || echo "run $i failed"; done; node scripts/flakiness.mjs /tmp/run-*.json

Expected output: A per-test pass/fail matrix across twenty runs names the nondeterministic tests with data instead of suspicion.

2. Move confirmed flaky tests to quarantine.

it("eventually syncs the order", { retry: false, tag: "@quarantine" }, async () => {
  // ...unchanged test body
});

Expected output: The main suite returns to a trustworthy red-or-green signal while the quarantined tests keep running and reporting.

3. Diagnose with the failure distribution.

node scripts/flakiness.mjs /tmp/run-*.json --test "eventually syncs" --show-seeds

Expected output: Failures clustering by time of day, order of siblings, or parallel load point at shared state, timing, or resource races respectively.

4. Fix the root cause, not the symptom.

// before: sleep-and-hope
await sleep(2000); expect(synced).toBe(true);

// after: await the actual condition
await waitFor(() => expect(synced).toBe(true), { timeout: 5000 });

Expected output: Polling the real condition removes the timing assumption; the test passes whenever the system is actually ready.

5. Graduate back after a clean window.

for i in $(seq 1 50); do npx vitest run -t "eventually syncs" >/dev/null 2>&1 || echo "still flaky"; done; echo done

Expected output: Fifty consecutive green runs earn the test its place back in the gating suite.

Troubleshooting

Problem

Quarantine becomes a graveyard

Solution

Set a time limit and an owner per quarantined test; anything older than two sprints gets fixed or deleted, not archived forever.

Problem

Cannot reproduce flakiness locally

Solution

The trigger is CI-specific concurrency or shared state; reproduce with the same shard count and parallelism flags, or log deterministically from CI runs.

Problem

Retry-on-failure hides the flakiness

Solution

Retries belong after the fix as defense in depth, not before it as a blindfold; measure flakiness with retries disabled.

Limitations

Flake detection by repetition is statistical; rare flakes need long windows or seeded-order experimentation.
Quarantine preserves signal for the rest of the suite but loses gating for the quarantined behavior until fixed.

A suite that lies sometimes lies always

One flaky test converts the whole suite into a probabilistic signal, and rerun-until-green trains the team to ignore red. The recovery is mechanical: run the suite repeatedly, name the nondeterministic tests with a failure matrix, quarantine them so the gating signal becomes honest again, then fix each by finding the assumption the test makes about time, order, or shared state.

# 20 runs, then a per-test pass/fail matrix
for i in $(seq 1 20); do
  npx vitest run --reporter=json > /tmp/run-$i.json
 done
node scripts/flakiness.mjs /tmp/run-*.json

The distribution names the cause

Flaky failures are not random even when they look it: failures that track wall-clock time implicate shared external state, failures that depend on neighboring tests implicate order coupling, and failures that scale with parallelism implicate resource contention. Reading the matrix before touching the test turns guesswork into a hypothesis.

Quarantine needs an expiry

Quarantine is an intensive care unit, not a retirement home. Every quarantined test gets an owner and a deadline; a test that stays past two sprints is either important enough to schedule the real fix or unimportant enough to delete. Without the expiry, the quarantine list silently becomes the suite everyone ignores, which is where flakiness began.