Shard Tests Across Parallel Databases for Speed

Give each test shard its own schema or database, run shards concurrently, and eliminate cross-test interference that serial runners hid.

~30 mintestingcidatabases

Task

Parallelize a database-bound test suite by sharding tests across isolated databases without flaky interference.

When to use this

The integration suite takes 25 minutes because every test shares one database and runs serially. A second schema is created occasionally, then rolled back wrongly and breaking everything after it.

Prerequisites

A test runner supporting shards or workers
A database where schemas or databases can be created per shard
CI able to run jobs in parallel

Steps

1. Provision one schema per shard in setup.

beforeAll(async () => {
  const shard = process.env.SHARD_INDEX ?? 0;
  const schema = `test_shard_${shard}`;
  await db.exec(`CREATE SCHEMA IF NOT EXISTS ${schema}`);
  await runMigrations(schema);
  process.env.DATABASE_URL = urlWithSchema(schema);
});

Expected output: Each shard owns a private schema with migrations applied, so identical tests never collide.

2. Run shards as parallel CI jobs.

strategy:
  matrix:
    shard: [1, 2, 3, 4]
  steps:
    - run: SHARD_INDEX=${{ matrix.shard }} SHARD_TOTAL=4 npm test -- --shard=${{ matrix.shard }}/4

Expected output: Four jobs each run a quarter of the suite against their own schema, wall-clock time drops accordingly.

3. Verify isolation by running the same test in all shards.

SHARD_INDEX=1 npm test -- -t "creates an order" & SHARD_INDEX=2 npm test -t "creates an order"; wait

Expected output: Both shards pass independently, proving state does not leak between schemas.

4. Tear down schemas deterministically.

afterAll(async () => {
  await db.exec(`DROP SCHEMA IF EXISTS ${schema} CASCADE`);
});

Expected output: CI leaves no shard residue, and reruns start clean regardless of prior failures.

5. Balance shards by historical duration.

node scripts/balance-shards.mjs --from .test-durations.json --shards 4

Expected output: Shards finish within seconds of each other, avoiding one long straggler defining the total time.

Troubleshooting

Problem

Tests still interfere across shards

Solution

Shared external state remains, such as a global Redis or a file path; isolate those per shard too, with the shard index in keys and paths.

Problem

Migrations run four times and slow setup

Solution

Create a template schema once and clone it per shard, which is far cheaper than replaying migrations per job.

Problem

One shard always fails late

Solution

It holds the longest tests; rebalance with recorded durations so each shard lands near the mean.

Limitations

Parallelism multiplies database load; ensure the CI database host can absorb shard-count times normal usage.
Per-shard schemas multiply connection counts; cap pool sizes per shard accordingly.

Isolation first, parallelism second

You cannot parallelize tests that share a database: the interference was invisible only because serialization hid it. Schemas per shard give every parallel lane a private world, and the suite splits into concurrently safe pieces. The discipline pays off beyond speed, because any test that only passed thanks to ordering becomes a visible failure you can actually fix.

// One private world per shard
const schema = `test_shard_${process.env.SHARD_INDEX}`;
await db.exec(`CREATE SCHEMA IF NOT EXISTS ${schema}`);
await runMigrations(schema);
process.env.DATABASE_URL = urlWithSchema(schema);

Order-dependence becomes visible

A suite that passes only in execution order is hiding coupling: test B depends on rows test A created. Sharding surfaces these instantly because the assumed predecessor may run in a different lane. Fixing them makes the suite deterministic under any ordering, which also removes a whole class of flakiness on reruns.

Balance beats equal counts

Splitting tests into equal counts leaves wall-clock time defined by the slowest shard, usually the one that inherited all the heavy integration cases. Balance by measured duration instead: record per-test times, bucket them near the mean, and the four shards finish together instead of one straggling for minutes.