A/B Test Significance Calculator

Each variant's conversion rate is an estimate with sampling error. Under the hypothesis that both variants really convert at the same rate, the difference between the observed rates divided by its standard error (using the pooled rate) follows a standard normal distribution; that ratio is z, and the two-sided p-value is the probability of a difference at least this large arising by chance. A result is called significant when p is below the level you set (5 % is conventional, not a law). The page also gives a confidence interval for the difference in percentage points — the range of true lifts consistent with the data.

“Significant” does not mean the observed uplift is the true one — the interval is usually wide — and “not significant” does not mean the variants are equal, only that the test could not tell. Checking the result every day and stopping when it looks good inflates false positives badly; decide the sample size first and read the result once. The normal approximation needs a handful of conversions in each group, and the page flags when there are too few.

Given the baseline rate, the smallest relative lift worth detecting, the significance level and the power (the chance of detecting that lift if it is real), the standard normal-approximation formula gives the visitors needed per variant. Smaller effects and lower baselines need far more traffic — halving the detectable lift roughly quadruples the sample. Other calculators use slightly different approximations and differ by a few percent. The normal distribution functions are computed in the page and checked against Python's reference implementation. Nothing leaves the browser; the same four anonymous usage counts as the rest of the site apply.