Test Quarantine
Test quarantine moves flaky tests out of the blocking CI path into a tracked, non-blocking lane with owners and exit criteria — containing nondeterminism instead of deleting or ignoring it.
Definition
Quarantine is the process of isolating failure-diagnosis-unreliable tests: they still run on a schedule and report results, but they no longer gate merges. Each quarantined test carries an owner, a root-cause hypothesis, and a time-boxed exit criterion — fixed and unquarantined, or deleted. The quarantine queue is a visible health metric, not a graveyard.
Context and scope
Applied in CI policy after a test flakes beyond a threshold. Works with retry-once policies and flake detection tracking pass/fail variance across runs.
Common misconceptions
- Quarantine means ignoring flaky tests — quarantined tests keep running and reporting; ignoring means silent deletion, which hides defects
- Retries make quarantine unnecessary — retries mask flakiness and slow every CI run paying for it; quarantine keeps the signal explicit
- The quarantine list is a backlog like any other — without owners and expiry it becomes a landfill where broken-but-real tests rot
Anti-patterns
- Quarantining without ownership or expiry and accumulating a permanent shadow suite
- Auto-retrying everything three times and calling the green runs truth
- Deleting flaky tests on sight because they annoy developers
Related concepts
Related examples
Authoritative source
What it is
A flaky test poisons the CI signal: teams learn to re-run on red, and real regressions slip through that habit. Quarantine removes the poison while preserving the test and the pressure to fix it.
How it works
Detection flags tests with bimodal outcomes across identical commits. Policy moves them to a non-blocking lane after a flake-rate threshold. The queue entry records owner, suspected cause — async races, order dependence, environment — and a deadline. Metrics: quarantine size trend and average residence time, both reviewed like any engineering health number.
When to apply
Immediately when a test flakes on identical code; before it trains the team to reflexively re-run. The fix usually involves removing the nondeterminism source: frozen clocks, seeded randomness, isolated state.
Pitfalls
Quarantine without exit criteria is denial with bookkeeping — enforce expiry by deletion. Over-quarantining hides systemic environment problems: many simultaneous flakes mean infrastructure, not tests. Retry policies interact badly: retrying quarantined tests defeats their diagnostic reporting.