Unmask Flaky Tests by Rerunning the Suite

Run the suite repeatedly with shuffled order and fixed seeds, rank tests by failure variance, and quarantine only the proven offenders.

~40 mintestingciflaky-tests

Task

Measure which tests fail intermittently across repeated runs, quarantine them with an expiry, and keep the suite trustworthy.

When to use this

CI fails randomly two or three times a week. The team reruns until green, and real regressions now hide inside the noise of ten known-flaky tests.

Prerequisites

A test runner supporting repeat or retry flags (vitest, jest, pytest)
CI time budget for at least three consecutive full runs
A quarantine mechanism: folder, tag, or config skip list

Steps

1. Run the suite three times in a row and capture every failure.

for i in 1 2 3; do npx vitest run --reporter=json > run-$i.json 2>/dev/null || true; done; for i in 1 2 3; do jq -r ".testResults[].assertionResults[] | select(.status == "failed") | .fullName" run-$i.json | sort > fail-$i.txt; done

Expected output: Three failure lists exist; diffing them shows which failures repeat and which flicker.

2. Compute which tests failed in some but not all runs.

cat fail-1.txt fail-2.txt fail-3.txt | sort | uniq -c | sort -rn | awk '$1 < 3 && $1 >= 1 {print $1, $2}' | head -15

Expected output: A ranked list of intermittent failures prints: count 1 or 2 out of 3 means flaky, count 3 means a real broken test.

3. Reproduce the flakiness deterministically with seeds and shuffled order.

npx vitest run --sequence.shuffle --sequence.seed=42 2>&1 | tail -5; npx vitest run --sequence.shuffle --sequence.seed=43 2>&1 | tail -5

Expected output: The same test fails under one seed and passes under another, proving order or timing dependence without guessing.

4. Quarantine proven offenders with an owner and an expiry date.

// vitest.quarantine.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    exclude: ['src/**/__tests__/**'],
    include: ['src/**/__tests__/**/*.test.ts'],
    // offenders listed explicitly, never silently
    testNamePattern: /^(?!.*(@flaky|@quarantine))/,
  },
});

// each quarantined test carries:
// it('checkout applies coupon [@flaky owner=amy expiry=2026-12-01 reason=redis timing]', ...)

Expected output: CI runs only the trustworthy suite; quarantined tests are visible in the repo with owners and expiry dates, not silently deleted.

5. Run the quarantine list separately so debt stays visible.

npx vitest run --testNamePattern "@flaky" 2>&1 | tail -8; ls src/**/__tests__/*quarantine* 2>/dev/null | wc -l

Expected output: A weekly job reports the quarantine pass rate; when a flaky test passes steadily again, it graduates back with a green run as evidence.

Troubleshooting

Problem

Failures never reproduce locally

Solution

CI-only flakiness usually stems from shared state or constrained resources; capture logs and timing, or run the suite inside the same container image locally to reproduce the environment.

Problem

Rerun-until-green is already habitual

Solution

Make retries visible: fail the build when a test needed a retry, and count reruns per week; the metric converts habit into pressure to fix root causes.

Problem

Quarantine list only ever grows

Solution

Attach an expiry to every entry and route expired items to an owner; a list without decay is a junk drawer, and deleting tests should stay a deliberate, reviewed act.

Limitations

Three runs prove intermittency only for observed seeds and orders; deeper timing bugs need stress loops or injected latency.
Quarantine hides signal from the main suite; the weekly report must stay honest about the pass rate inside the quarantine lane.

Rerun-until-green is how regressions go into hiding

When a red build is routinely retried into green, the suite stops being evidence and becomes noise. Three consecutive runs, a variance count, and a seed-shuffle reproduction turn the vague feeling of flakiness into a ranked, owned, time-boxed list.

# Three runs, three failure lists
for i in 1 2 3; do
  npx vitest run --reporter=json > run-$i.json || true
done

# Which tests fail sometimes, and which always?
cat fail-1.txt fail-2.txt fail-3.txt | sort | uniq -c
# 3  checkout/applies-coupon      -> genuinely broken
# 1  search/debounces-input       -> flaky (order/time dependent)
# 2  auth/refresh-token           -> flaky (clock skew)

# Reproduce one offender deterministically
npx vitest run --sequence.shuffle --sequence.seed=42

Variance is the flakiness fingerprint

A test failing in all three runs is broken and gets fixed. A test failing in one or two runs is flaky, and the count is the ranking: fail-in-two beats fail-in-one for triage priority. This single aggregation replaces hours of arguing about which test someone half-remembers seeing fail.

Quarantine with decay, or do not bother

A quarantine without expiry and owners is where tests go to be forgotten. Each entry needs a reason, a person, and a date. The separate weekly run keeps the debt visible and measurable: when the quarantine pass rate hits 100 percent for two weeks, tests graduate back.