Flaky Test
A flaky test passes and fails on the same code, eroding trust in the suite and masking real regressions.
Definition
A flaky test produces different results across runs without any change to the code under test. The common causes are timing and race conditions, dependence on shared or external state, non-deterministic ordering, and insufficient waiting for asynchronous work. Flakiness is costly not because a single failure matters, but because it trains teams to re-run and ignore failures, so genuine regressions slip through the noise.
Context and scope
Prevalent in suites that touch time, concurrency, networks, and databases, and in large CI fleets where random failures accumulate quickly.
Common misconceptions
- Retrying makes flaky tests go away — retries hide the symptom and let a real intermittent defect stay in production
- Flakiness is inevitable at scale — most flakes have identifiable causes; tolerating them as normal is a choice with consequences
- The test is always the problem — sometimes the test exposes a real race in the code that only appears intermittently
Anti-patterns
- Marking a flaky test as skipped forever instead of fixing or quarantining it with a deadline
- Auto-retrying failures until they pass, with no tracking of how often retries are needed
- Blaming infrastructure so consistently that no one investigates individual flakes
Related concepts
Related examples
Authoritative source
What it is
A flaky test is non-deterministic: identical code yields pass on one run and fail on the next. The usual culprits are shared state leaking between tests, dependence on real time or ordering, insufficient synchronisation when waiting for asynchronous results, and external systems that are themselves variable. The symptom is intermittent failure; the causes are concrete and, in most cases, findable.
Why it matters
A suite that fails at random cannot be trusted, so teams learn to re-run until green and to dismiss red builds as noise. That habit is exactly what lets a real regression in. Flakiness also slows delivery, burns CI capacity, and trains engineers to distrust their own tests, which is a cultural cost far larger than any single failure.
How to use it well
Make tests deterministic: control time with an injectable clock, isolate state in fixtures, avoid ordering dependencies, and wait on conditions rather than fixed sleeps. For known-flaky external dependencies, introduce fakes. When a flake appears, quarantine it with an owner and a deadline rather than deleting it, and analyse retry statistics to find the worst offenders. Above all, treat a flaky test as a defect in the test or the code, not as an acceptable cost of scale.