Testing Haven

A/B Test Significance Calculator

Is your test result statistically significant?

Test data

Control

Variant B

Test type(two-tailed detects changes in either direction — the safer default)

Enter visitors and conversions for the control and at least one variant to see results.

What this calculator does

This A/B test significance calculator tells you whether the difference in conversion rate between your control and one or more variants is statistically significant — that is, whether it is large enough, given your sample size, that random chance is an unlikely explanation. You enter visitors and conversions for each group; it returns the conversion rates, the relative lift, a p-value from a two-proportion z-test, a confidence interval on the lift, and a plain-English verdict you can paste straight into Slack.

Every input is encoded in the URL, so you can share a link to your exact result with your team, and export the result card as a PNG for decks and readouts.

The formula

The calculator runs a two-proportion z-test. With n1,n2n_1, n_2 visitors and observed conversion rates p^1,p^2\hat{p}_1, \hat{p}_2 for control and variant, the test statistic is:

z=p^2−p^1p^(1−p^)(1n1+1n2),p^=x1+x2n1+n2z = \frac{\hat{p}_2 - \hat{p}_1}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}, \qquad \hat{p} = \frac{x_1 + x_2}{n_1 + n_2}

where x1,x2x_1, x_2 are the conversion counts and p^\hat{p} is the pooled rate under the null hypothesis that both groups convert identically. The two-tailed p-value is p=2 (1−Φ(∣z∣))p = 2\,(1 - \Phi(|z|)), where Φ\Phi is the standard normal CDF. The confidence interval on the difference uses the unpooled standard error, and the interval on relative lift uses the delta method.

A worked example

Suppose your control had 10,000 visitors and 1,000 conversions (10.0%), and your variant had 10,000 visitors and 1,120 conversions (11.2%). The pooled rate is 10.6%, giving a standard error of 0.435 percentage points and z=2.75z = 2.75. The two-tailed p-value is 0.006 — well below 0.05 — so the 12% relative lift is statistically significant at the 95% (and even 99%) confidence level. The 95% confidence interval on the relative lift runs from roughly +3.5% to +20.5%: the true effect is very likely positive, but its size is still uncertain.

When to use it

  • After your test has reached its planned sample size (use the sample size calculator before launch).
  • For binary metrics: conversion, sign-up, click-through. For revenue or other continuous metrics, use the revenue-per-visitor calculator.
  • For 2+ variants: add variants and the calculator applies an omnibus chi-square test plus Holm-corrected pairwise comparisons automatically.

Common mistakes

  • Peeking. Checking daily and stopping when p first dips below 0.05 inflates false positives dramatically. Decide the sample size in advance, or use the sequential testing calculator, whose p-values remain valid under continuous monitoring.
  • One-tailed tests chosen after the fact. Halving the p-value by switching to one-tailed after seeing a positive result is p-hacking. Default to two-tailed.
  • Ignoring sample ratio mismatch (SRM). If your 50/50 test shows a 52/48 split of actual traffic, your assignment may be broken and every p-value in the test suspect. Run the SRM checker first.
  • Multiple comparisons without correction. Each extra variant or metric multiplies your chances of a fluke. Use the built-in Holm correction for A/B/n tests.
  • Confusing significance with size. With enough traffic, a trivial +0.2% lift can be "significant". Read the confidence interval, not just the verdict.

Frequently Asked Questions

What p-value counts as statistically significant?

By convention, a result is called statistically significant when the p-value is below 0.05, which corresponds to a 95% confidence level. This threshold is a trade-off, not a law of nature: at p < 0.05 you accept a 5% chance of declaring a winner when there is no real difference. For high-stakes or hard-to-reverse changes, many teams require p < 0.01 instead.

Should I use a one-tailed or two-tailed test?

Use a two-tailed test unless you have a specific, pre-registered reason not to. A two-tailed test can detect both improvements and regressions. A one-tailed test reaches significance faster, but it is blind to the possibility that your variant is worse — and switching to one-tailed after seeing the data is a form of p-hacking that inflates your false positive rate.

My test has been running for a week and p is 0.06. Can I keep it running until it hits 0.05?

Only if you planned the longer duration in advance. Repeatedly checking a fixed-horizon test and stopping the moment p dips below 0.05 (called "peeking") badly inflates the false positive rate — with daily checks, a nominal 5% error rate can become 20% or more. Either fix the sample size in advance with a sample size calculator, or use a sequential testing method with always-valid p-values.

How is this different from the Bayesian calculator?

This calculator answers the frequentist question: "if there were truly no difference, how surprising would this data be?" The Bayesian calculator answers: "given this data, what is the probability that the variant is better, and how much do I risk by shipping it?" Both are valid frameworks; the Bayesian output is often easier to act on, while p-values remain the standard language of experimentation programs.

What happens when I test more than one variant (A/B/n)?

Each extra variant adds another comparison, and every comparison carries its own false positive risk. Testing 4 variants at α = 0.05 gives roughly an 18.5% chance of at least one false positive if you do not correct for it. This calculator runs an omnibus chi-square test across all groups and adjusts pairwise p-values with the Holm (default) or Bonferroni correction to keep the overall error rate at your chosen α.

Do visitors and conversions need to be unique counts?

Yes — the test assumes each visitor is an independent trial that either converted or did not. Use unique visitors (or users, or sessions — consistently) and count each converter once. Counting total conversion events, where one user can convert repeatedly, violates the independence assumption and makes the p-value unreliable.