Flaky Tests
A flaky test passes and fails on the same code, eroding trust in the suite; the fix is finding the nondeterminism, not rerunning until green.
Definition
A flaky test is a test whose outcome changes between runs without any change to the code or environment it is meant to verify.
Context and scope
Common in suites that depend on timing, shared state, network access, randomness, or execution order, and especially costly in continuous integration.
Common misconceptions
- Flaky tests are an unavoidable cost of testing at scale
- Automatic retries are a sufficient fix
- A flaky test is harmless as long as it usually passes
Anti-patterns
- Rerunning failed jobs until they pass and then merging
- Quarantining tests with no owner or expiry date
- Adding fixed sleeps to hide a race condition
Related concepts
Related examples
Authoritative source
Why flakiness matters
A test that passes locally and fails in CI, or passes on rerun, teaches the team that red does not mean broken. People start ignoring failures, merging through red builds, and rerunning until green. The suite loses its ability to block bad changes, and real defects hide among the noise. Flakiness is therefore a reliability problem in the delivery system, not a minor inconvenience.
Root causes
Most flakiness comes from uncontrolled state or time. Tests share a database row, a cache key, or a global singleton. They assume a fixed clock, a timezone, or an ordering that the runtime does not guarantee. Asynchronous work finishes after the assertion. Network calls time out intermittently. Random data occasionally hits an edge case. Test order changes because a runner parallelizes files. The defect may be in the test, the production code, or the environment.
Detection
Track outcomes per test across runs instead of only the latest result. Flag tests that fail and then pass on the same commit, or whose failure rate is nonzero over many runs. Repeat suspicious tests in a loop with different seeds and parallelism. Record the seed, environment, and logs so a failure can be reproduced. Dashboards that rank flaky tests by lost engineering time make the cost visible.
Repair
Reproduce first, then fix the cause: inject a clock, isolate data, await the actual signal instead of sleeping, seed randomness, or remove order dependence. If the production code has a race, fix the code. Retries are a mitigation, not a repair; compared to fixing the race, they also hide how often it occurs. Quarantine is acceptable only with an owner and a deadline.
Policy
For example, a team can require that a flaky test be fixed or deleted within ten working days. Deleting a test that checks nothing reliable is better than keeping a lying signal.