Test an A/B Result for Statistical Significance

Background

Shipping on a small uplift that is noise destroys trust in experiments. A stated threshold with computed evidence separates a result from a guess.

Problem

Compute the relative lift and a two-proportion z-score from control and variant conversions, and flag significance at the two-sigma threshold.

Input

JSON object: { control: { visitors, conversions }, variant: { visitors, conversions } }

Output

JSON object: { liftPercent, zScore, significant }

Sample tests

Input: {"control":{"visitors":1000,"conversions":100},"variant":{"visitors":1000,"conversions":100}}

Expected: {"liftPercent":0,"zScore":0,"significant":false}

Input: {"control":{"visitors":5000,"conversions":250},"variant":{"visitors":5000,"conversions":300}}

Expected: {"liftPercent":20,"zScore":2.193,"significant":true}

Your solution

Learning objective

Decide experiment outcomes with a stated statistical bar.