阅读中文版本

Test Flakiness Metrics

Flakiness metrics quantify how often a test passes and fails on unchanged code, turning a vague feeling into a ranking for repair.

Definition

Test flakiness metrics are measurements that estimate the probability a given test produces inconsistent results across otherwise identical runs, typically from repeated execution of the same revision.

They are used to prioritise which tests to fix or quarantine so that trust in the suite does not erode. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.

When to use it

A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.

How it works

Repeat-running the same commit separates deterministic failures from nondeterministic ones, and the flip rate per test ranks candidates for investigation.

A flaky suite trains developers to ignore red builds, which hides genuine regressions, so flakiness is a reliability problem rather than a minor annoyance.

Trade-offs and alternatives

Fixing root causes such as timing assumptions, shared state, or uncontrolled clocks is durable; automatic retries merely mask the symptom. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.

Common misconceptions

A persistent misconception is that a single retry pass proves a test is stable. Another is that flaky tests can be ignored because they always eventually pass. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.

Anti-patterns to avoid

Two recurring anti-patterns are to quarantining flaky tests and never repairing them and to rerunning failed tests automatically without tracking the rate. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include Accessibility Tree Testing, The Test Oracle Problem, Branch Coverage.