Test Order Dependence
Test order dependence is hidden coupling between tests — one test leaves state another consumes — surfacing only when the runner shuffles or parallelizes and the suite turns red at random.
Definition
Order dependence exists when tests are not independent: earlier tests seed, mutate, or leak state — database rows, singletons, cached modules, filesystem artifacts — that later tests implicitly rely on. Sequential alphabetical execution masks it; random ordering, parallel shards, and CI differences expose it. Detection runs the suite in multiple permutations and diffs failures.
Context and scope
Found via shuffle plugins (pytest-randomly, Jest random order), parallel CI sharding, and failure clustering that follows specific test pairs rather than specific tests.
Common misconceptions
- A suite passing locally is correct — local runs usually repeat one fixed order; the coupling waits for the first reorder to strike
- Retries fix order dependence — retries mask it; the coupling remains and fails unpredictably under parallelization
- Only shared databases cause it — module-level singletons, class reloads, environment variables, and clock state couple tests too
Anti-patterns
- Disabling random ordering after the first failures and entrenching the hidden dependency graph
- Designing test A to run before test B by convention and shipping an invisible graph nobody can safely parallelize
- Resetting singletons in some tests but not others, so the suite works only in the order that happens to reset
Related concepts
Related examples
Authoritative source
What it is
The suite is not a set of independent experiments but an accidental pipeline. Each test assumes the residue of its predecessors. Any change to execution order — sharding, shuffle, new test inserted alphabetically — invalidates the assumptions.
How it works
Coupling channels: shared database state, module and class-level mutable state, environment and configuration set by earlier tests, and filesystem artifacts. Detection runs permutations — random order plugins flag pairings; bisect mode isolates the minimal failing sequence. Fixes go to the isolation layer: per-test transactions, fresh module imports, explicit reset fixtures.
When to hunt
Before enabling parallelization, after adding retry policies, and whenever failures cluster on specific orders rather than specific tests. Randomized order in CI on a schedule keeps coupling from regrowing.
Pitfalls
The first reflex — reorder to the working sequence — converts an invisible graph into policy. Inter-test dependencies encode as comments and conventions, which refactors silently violate. Parallel shards expose coupling nondeterministically: the same pair of tests collides only when scheduled together, producing the most confusing flake signature.