A/B Test Statistical Significance
A/B significance is the discipline of separating real effects from noise: fixed hypothesis, pre-computed sample size, guardrail metrics, and no peeking before the planned horizon.
Definition
Statistical significance in controlled experiments is the probability that an observed difference reflects a real effect rather than sampling noise. Sound practice fixes the hypothesis and metrics before launch, computes required sample size from baseline variance and minimum detectable effect, avoids optional stopping, and applies multiple-comparison corrections across metric families.
Context and scope
Applied on product experiments and performance rollouts. Tools compute power analysis; organizations encode the discipline in experiment review checklists.
Common misconceptions
- A p-value below 0.05 proves the feature works — it suggests data compatible with an effect at that threshold, one time in twenty by construction
- Peeking daily and stopping when significant is efficient — optional stopping inflates false positive rates dramatically; sequential methods exist for early looks
- More metrics mean more confidence — every added comparison multiplies the family-wise error rate unless corrected
Anti-patterns
- Declaring winners after three days on a test powered for three weeks
- Reading every dashboard metric and reporting whichever reached significance
- Running five overlapping experiments on the same population without accounting for interaction
Related concepts
Related examples
Authoritative source
What it is
Randomization creates two populations differing only by treatment. Noise makes them differ anyway. Significance machinery quantifies how surprising the observed gap would be under the no-effect hypothesis, so decisions rest on evidence rather than enthusiasm.
How it works
Pre-register: primary metric, direction, minimum detectable effect, significance level. Power analysis yields the sample size and duration — including full weekly cycles to absorb day-of-week effects. Evaluate at the horizon with the fixed analysis; apply Bonferroni or false discovery rate corrections when scanning metric families; check guardrails for harm before shipping any win.
When to enforce
Any decision justified by data: UI changes, algorithm tuning, infrastructure cost changes with user-visible risk. Encode the checklist in the experiment platform so violations need explicit overrides.
Pitfalls
Sample ratio mismatch — unequal group sizes — reveals instrumentation bugs and invalidates the run. Novelty effects fade after weeks; holdout cohorts measure durability. Interfering experiments violate the isolation assumption silently. Tiny effects declared from huge samples are statistically real and practically meaningless — report effect size with intervals, not significance alone.