Mutation Testing
Mutation testing measures test quality by injecting small code changes — flipping operators, removing calls — and checking whether the tests catch the mutant, exposing tests that pass without.
Definition
Mutation testing is a technique where the source under test is systematically mutated — changing a plus to a minus, removing a conditional, altering a return — and the test suite is run against each mutant; mutants that the tests do not kill reveal tests that pass regardless of the behavior, i.e., tests with inadequate assertions.
Context and scope
Applied in test quality assessment, coverage gap analysis, and any context where line coverage is high but defect detection is suspected to be weak.
Common misconceptions
- High coverage means the tests are good — coverage measures which code ran, not whether the tests would fail if the code changed; mutation testing measures the latter
- Mutation testing is too slow for regular use — incremental and selective mutation make it tractable for CI; running every mutant every time is the slow path
Anti-patterns
- Using mutation testing as a gate without excluding equivalent mutants, producing unkillable failures that block the build
- Mutating only one line and declaring the suite strong when the tests catch it, missing the untested branches
Related concepts
Authoritative source
What it is
Mutation testing is a technique where the source under test is systematically mutated — changing a plus to a minus, removing a conditional, altering a return — and the test suite is run against each mutant; mutants that the tests do not kill reveal tests that pass regardless of the behavior, i.e., tests with inadequate assertions. Applied in test quality assessment, coverage gap analysis, and any context where line coverage is high but defect detection is suspected to be weak. The distinguishing question is what counts as evidence: a test that passes by accident is worse than no test, because it manufactures false confidence that propagates into release decisions.
How it works
Sound tests exercise a specific behavior contract, isolate the unit under test from environmental noise, and fail with a diagnostic that points at the cause rather than the symptom. Determinism is engineered, not assumed: clocks are fixed, randomness is seeded, and external state is replaced with doubles whose interactions are asserted, not merely stubbed. Coverage measures which behaviors were exercised, not which lines were visited.
When to apply and pitfalls
Apply where the cost of an undetected regression exceeds the cost of writing and maintaining the test. The recurring pitfall is testing the implementation rather than the behavior — such tests break on every refactor and teach nothing. Equally damaging are tests that assert only the happy path, leaving the error branches — where most production bugs live — unexercised.