Prove Test Isolation with Random Order Execution

Run the suite under shuffled orders and multiple seeds, identify tests that leak state into siblings, and fix the leak at the source.

~35 mintestingisolationci

Task

Detect order-dependent tests with shuffle-plus-seed runs, trace the leaked state to its producer, and eliminate the coupling.

When to use this

The suite is green on CI but fails locally when run as a single file. A test depends on database rows another test created, and nobody knows which.

Prerequisites

A runner with shuffle and seed flags (vitest --sequence.shuffle)
A local environment matching CI versions
Freedom to refactor setup and teardown code

Steps

1. Establish the failure depends on order, not content.

npx vitest run tests/orders.test.ts tests/auth.test.ts 2>&1 | tail -3; npx vitest run tests/auth.test.ts tests/orders.test.ts 2>&1 | tail -3

Expected output: One order passes and the reverse fails, proving state leaks from one file into the other via a shared resource.

2. Run the full suite shuffled across several seeds to measure the blast radius.

for s in 11 22 33 44; do echo "seed=$s"; npx vitest run --sequence.shuffle --sequence.seed=$s 2>&1 | grep -E "^.*(Tests|Test Files).*(passed|failed)" | head -2; done

Expected output: Failure counts vary by seed; seeds failing with zero failures mark the lucky orders that have been hiding the leak.

3. Bisect the leak with the failing seed and filtered files.

npx vitest run --sequence.shuffle --sequence.seed=22 tests/auth.test.ts tests/orders.test.ts 2>&1 | grep -E "×|✓" | head -10

Expected output: The failing test under the specific seed prints; it now has a producer (its predecessor in that order) and a consumer, both named.

4. Find the leaked state in the consumer by asserting emptiness first.

// consumer test: assert the world is clean before acting
it('lists orders for the current tenant', async () => {
  const before = await db.count('orders', { tenant: 't1' });
  expect(before).toBe(0);            // <-- fails under leak, naming the leak
  await createOrder({ tenant: 't1' });
  const rows = await listOrders('t1');
  expect(rows).toHaveLength(1);
});

Expected output: The added precondition fails under the shuffled order and passes in isolation, converting a mysterious sibling failure into a precise leak detector.

5. Fix the producer with real teardown, then prove order independence.

npx vitest run --sequence.shuffle --sequence.seed=22 2>&1 | tail -2 && npx vitest run --sequence.shuffle 2>&1 | tail -2

Expected output: The previously failing seed passes, and an unseeded shuffle also passes, establishing order independence as a durable property.

Troubleshooting

Problem

Failures point at random tests each seed

Solution

Multiple independent leaks exist; fix them one seed at a time, starting with the consumer whose precondition assertion fails, since each fix shrinks the failure surface.

Problem

Shuffle is too slow on a large suite

Solution

Shuffle nightly or per pull request rather than every save; combine with file-level parallelism so wall time stays comparable.

Problem

The leak lives in a singleton outside the test files

Solution

Module-level caches and connection pools outlive files; expose a reset hook and call it in global setup or afterAll, or scope singletons per worker with fresh instantiation.

Limitations

Shuffling proves isolation for observed orders; rare interleavings still need explicit concurrency testing or stress loops.
Seeds vary by runner version; record the runner version alongside the seed when filing reproducible order bugs.

Green in order is not green, it is lucky

A suite that only passes in its habitual order is a suite whose tests share state invisibly. Shuffled execution with recorded seeds turns luck into a property you can test: either every order passes, or a specific leak has a name, a producer, and a fix.

# Reverse the usual order -- does it still pass?
npx vitest run tests/auth.test.ts tests/orders.test.ts
# FAILS: orders lists 3 rows, expected 1

# Reproduce with a recorded seed
npx vitest run --sequence.shuffle --sequence.seed=22
# × orders > lists orders for the current tenant

# After fixing the producer teardown
npx vitest run --sequence.shuffle --sequence.seed=22   # all pass
npx vitest run --sequence.shuffle                      # all pass, any order

Preconditions are leak detectors

The fastest way to find who leaked is to make the victim testify: start each test with an assertion that the world is empty. A clean suite passes these trivially. A leaking suite fails at exactly the line where foreign state first becomes visible, which localizes the search better than any debugger session.

Fix the producer, not the consumer

The tempting fix is deleting rows at the start of the failing test, which masks the leak while leaving it live. The real fix is teardown in the test that created the rows, or per-test fixtures that guarantee a fresh scope. Consumers should be allowed to assume a clean world; that assumption is what isolation means.