Change-Based Regression Test Selection Workflow
Select regression tests by code-change to test mapping, run the full suite nightly, and validate selection recall against escaped defects.
Objective
Cut pre-merge regression time by running the tests that the change can affect, with nightly full runs and measured recall keeping the shortcut honest.
Environment
- A mapping from source files to tests (coverage data or static import analysis)
- CI capable of running selected subsets
- A nightly full-suite slot
Steps
1. Build the change-to-test map
Derive which tests exercise which source files from coverage runs or static analysis, storing the mapping as a versioned artifact.
Expected state: A queryable map exists: given changed files, it returns the candidate test set.
2. Select and run on PRs
For each PR, run the tests mapped to its changed files plus a fixed smoke set, reporting selection size versus suite size.
Expected state: PR regression time drops to the selected set while remaining proportional to the change.
3. Run the full suite nightly
Keep a nightly full run as the safety net, with failures triaged by whether the selecting PR should have caught them.
Expected state: Nightly failures classified as selection-miss feed the map's improvement.
4. Measure recall on escaped defects
For every defect that reached nightly or production, check whether the mapping would have selected a failing test at the PR stage.
Expected state: A recall metric exists: percentage of escapes that selection would have caught.
5. Tighten the map where recall fails
For each selection-miss, add the missing edge (file to test) via explicit dependency or integration-test tagging, growing coverage of the map itself.
Expected state: Recall improves monotonically, and the map's blind spots are known and shrinking.
Expected outcome
Fast PR feedback through change-based selection with nightly full runs, and a measured recall that justifies the shortcut with evidence.
Verification
PR regression time drops substantially; nightly escapes trend down; the recall metric on recent defects exceeds the agreed threshold.
Run what the change can touch, prove what you skipped
Running every test on every PR is safe and slow; the shortcut is selection by change-to-test mapping, made honest by nightly full runs and a recall metric on escaped defects. The map earns trust by demonstrating what it would have caught.
PR #4821 changed: api/checkout.ts, lib/pricing.ts
map lookup:
api/checkout.ts -> [checkout.e2e, api.contract,
promo.unit] (3 tests)
lib/pricing.ts -> [pricing.unit, promo.unit]
selected: 4 tests + smoke (11) suite: 1,240
PR time: 38 min -> 3.4 min
nightly full: 1 failure (tax/calc.ts)
escaped from PR #4808 (touched tax/calc.ts)
selection-miss: map lacked tax/calc -> tax.e2e edge
-> edge added; recall 87% -> 91%Recall is the license to select
Selection without recall measurement is just skipping tests with confidence. Every defect that escapes to nightly or production is a free experiment: would the map have selected a failing test at the PR? The answer, tracked over time, is the number that justifies the entire approach.
Integration seams are the map’s blind spot
File-to-test edges derived from coverage miss exactly the class of regression that matters most: a change in one module breaking behavior composed across modules. Explicit tagging of integration tests to the seams they guard is how recall failures get repaired.