Benchmark Variance Reduction
Benchmark variance reduction controls the noise sources — CPU frequency scaling, thermal throttling, scheduler interference — that make a single benchmark run unrepeatable and a comparison.
Definition
Benchmark variance reduction is the set of experimental controls — pinning CPU frequency, disabling turbo boost, isolating cores, warming the JIT, running enough iterations for statistical power — that make benchmark measurements repeatable enough that a difference between runs reflects a performance change rather than environmental noise.
Context and scope
Applied in performance regression suites, microbenchmark comparison, and any measurement where the change to detect is smaller than the noise floor of the machine.
Common misconceptions
- More iterations fix variance — they reduce sampling error but not systematic bias from throttling or scheduling; both must be controlled
- A statistically significant difference is a real difference — it is real for this machine in this state; generalizing requires controlled conditions and repetition
Anti-patterns
- Comparing a single before run to a single after run and reporting the delta as the improvement
- Running benchmarks on a shared CI machine where neighbor noise dominates the measurement
Related concepts
Authoritative source
https://benchmarkdotnet.org/articles/guides/good-practices.html
What it is
Benchmark variance reduction is the set of experimental controls — pinning CPU frequency, disabling turbo boost, isolating cores, warming the JIT, running enough iterations for statistical power — that make benchmark measurements repeatable enough that a difference between runs reflects a performance change rather than environmental noise. Applied in performance regression suites, microbenchmark comparison, and any measurement where the change to detect is smaller than the noise floor of the machine. The distinguishing question is what counts as evidence: a test that passes by accident is worse than no test, because it manufactures false confidence that propagates into release decisions.
How it works
Sound tests exercise a specific behavior contract, isolate the unit under test from environmental noise, and fail with a diagnostic that points at the cause rather than the symptom. Determinism is engineered, not assumed: clocks are fixed, randomness is seeded, and external state is replaced with doubles whose interactions are asserted, not merely stubbed. Coverage measures which behaviors were exercised, not which lines were visited.
When to apply and pitfalls
Apply where the cost of an undetected regression exceeds the cost of writing and maintaining the test. The recurring pitfall is testing the implementation rather than the behavior — such tests break on every refactor and teach nothing. Equally damaging are tests that assert only the happy path, leaving the error branches — where most production bugs live — unexercised.