Statistical Significance in A/B Testing, Explained

"Is it significant?" is the most-asked and most-misunderstood question in A/B testing. This guide explains what statistical significance actually claims, what it doesn't, and the handful of mistakes that quietly turn "significant" results into fiction.

What a p-value actually says

The p-value answers one narrow question: if the variant truly changed nothing, how likely is data at least this extreme? A p-value of 0.03 means that a world with no real difference would produce a gap this large only 3% of the time. That makes the no-difference explanation implausible — which is all "significant" means.

Note what the p-value does not say. It is not the probability the variant is better (that is the Bayesian question — see the Bayesian calculator). It is not the size of the effect. And 1 − p is not your "chance of winning". A tiny p-value with a huge sample can accompany a lift too small to matter; always read the confidence interval alongside it.

The machinery: the two-proportion z-test

For conversion metrics, significance is usually computed with a two-proportion z-test: the observed difference in rates divided by the standard error you would expect under the null hypothesis. The result (z) is converted into the p-value. Our significance calculator runs this test, shows the confidence interval on the lift, and — for A/B/n tests — applies multiple-comparison corrections automatically.

The three mistakes that fake significance

1. Peeking. Checking daily and stopping the moment p < 0.05 turns a 5% false-positive rate into 20%+. The fix is either a pre-committed sample size (use the sample size calculator) or an always-valid method (the sequential calculator).

2. Multiple comparisons. Four variants, five metrics, three segments = 60 chances for a fluke. Corrections like Holm's method keep the family-wise error rate honest; our significance calculator applies them for multi-variant tests.

3. Broken randomization. If the traffic split itself is off — a sample ratio mismatch — the p-value is answering a question about corrupted data. Run the SRM checker before believing any result.

Reading a significant result like an adult

A significant result earns three follow-up questions. Was the test adequately powered (an underpowered win exaggerates the true effect — check the power calculator)? Does the confidence interval include effect sizes you would not ship? And does the result survive the data-quality checks? If yes to all three, ship it — and expect the measured lift to shrink a little in production. That is not failure; that is regression to the mean.