A/B Experiment Testing Workflow
State the hypothesis and metric, define the sample and duration, guard against peeking, run the experiment, and act on the result honestly.
Objective
Run a controlled A/B experiment that produces a trustworthy decision.
Environment
- An experimentation platform
- A primary metric with sufficient traffic
Steps
1. State the hypothesis and the single
State the hypothesis and the single primary metric.
Expected state: The decision criteria are fixed upfront.
2. Compute the required sample and duration
Compute the required sample and duration in advance.
Expected state: The experiment can reach significance.
3. Avoid peeking and stopping early
Avoid peeking and stopping early on noise.
Expected state: Results are not p-hacked.
4. Analyze the result, including guardrail metrics,
Analyze the result, including guardrail metrics, and decide.
Expected state: The decision considers harm, not only the win.
Decision points
Stop early on a big lift?
- No, pre-register
- Yes
Do not stop early; sequential stopping without correction inflates false positives.
Ship on a flat result?
- Only if it helps a guardrail
- No
A flat primary metric may still justify shipping if it improves a guardrail such as speed.
Expected outcome
A trustworthy experiment result with a documented decision to adopt, reject, or iterate.
Verification
Confirm the pre-registered sample size was reached, guardrail metrics were within bounds, and the decision followed the pre-stated criterion.
Cleanup
Archive the experiment configuration and result, and clean up the treatment code.
Overview
Run a controlled A/B experiment that produces a trustworthy decision. Confirm the pre-registered sample size was reached, guardrail metrics were within bounds, and the decision followed the pre-stated criterion.
Steps
- State the hypothesis and the single primary metric. Expected state: The decision criteria are fixed upfront.
- Compute the required sample and duration in advance. Expected state: The experiment can reach significance.
- Avoid peeking and stopping early on noise. Expected state: Results are not p-hacked.
- Analyze the result, including guardrail metrics, and decide. Expected state: The decision considers harm, not only the win.
const variant = hash(userId) % 100 < 50 ? 'B' : 'A';
track('experiment_exposure', { experiment: 'checkout-cta', variant });Verification
- A trustworthy experiment result with a documented decision to adopt, reject, or iterate.
Decision points
- Stop early on a big lift? Options: No, pre-register; Yes. Recommended: Do not stop early; sequential stopping without correction inflates false positives.
- Ship on a flat result? Options: Only if it helps a guardrail; No. Recommended: A flat primary metric may still justify shipping if it improves a guardrail such as speed.