Skip to content
TESTING HAVEN

II · ANALYZE

A/B Test Significance Calculator

Two schools of statistics. One clear verdict.

An A/B test result is statistically significant when the gap between the variants is too large to put down to chance. For example, 10,000 visitors per group converting at 10.0% and 11.2% is a +12.0% lift with p = 0.0058: significant at 95% confidence. This calculator gives that reading and the Bayesian one side by side.

Control — the original

Variant B — the challenger

Test type
ADVANCED — PRIORS, RISK THRESHOLD & TRAFFIC SPLIT
%

Expected traffic split for the SRM check. Leave blank for an even split.

Enter visitors and conversions for the control and at least one variant — the frequentist and Bayesian analyses run side by side automatically.

How the mathematics works

What this calculator does

This A/B test calculator evaluates your experiment two ways at once. You enter visitors and conversions for each group, and it runs a frequentist significance test and a Bayesian analysis side by side: p-values, confidence intervals, and lift on the left; probability to be best, expected loss, credible intervals, and the full posterior probability distribution of each group's conversion rate on the right. The two frameworks answer different questions about the same data, and seeing them together tells you far more than either alone.

It also runs a sample ratio mismatch (SRM) check on your visitor counts automatically — if your traffic split is broken, no amount of statistics can rescue the test, so the calculator warns you before you read anything else.

Every input is encoded in the URL, so you can share a link to your exact result with your team, and export the result card as a PNG for decks and readouts.

The frequentist test

The calculator runs a two-proportion z-test. With n1,n2n_1, n_2 visitors and observed conversion rates p^1,p^2\hat{p}_1, \hat{p}_2 for control and variant, the test statistic is:

z=p^2−p^1p^(1−p^)(1n1+1n2),p^=x1+x2n1+n2z = \frac{\hat{p}_2 - \hat{p}_1}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}, \qquad \hat{p} = \frac{x_1 + x_2}{n_1 + n_2}

where x1,x2x_1, x_2 are the conversion counts and p^\hat{p} is the pooled rate under the null hypothesis that both groups convert identically. The two-tailed p-value is p=2 (1−Φ(∣z∣))p = 2\,(1 - \Phi(|z|)), where Φ\Phi is the standard normal CDF. The confidence interval on the difference uses the unpooled standard error, and the interval on relative lift uses the delta method.

Two refinements keep the result honest at the edges. When any group is expected to have fewer than 5 conversions (or non-conversions), the normal approximation behind the z-test is unreliable, so the p-value comes from Fisher's exact test instead, which is conservative. And a one-tailed test reports a one-sided lower bound on the lift at your confidence level, not a narrower two-sided interval.

The Bayesian model

The Bayesian panel treats each group's true conversion rate as an uncertain quantity with a Beta prior (uniform Beta(1, 1) by default, configurable under Advanced). Observing xx conversions in nn visitors gives the closed-form posterior

p∣data  ∼  Beta(α0+x,  β0+n−x)p \mid \text{data} \;\sim\; \mathrm{Beta}(\alpha_0 + x,\; \beta_0 + n - x)

which is exactly the probability distribution drawn for each group in the posterior chart. From those posteriors the calculator computes, by numerical integration, the probability that a variant beats control and the expected loss of shipping it:

Pr⁡(pB>pA)=∫01FA(x) fB(x) dx,E[max⁡(pA−pB, 0)]=∫01FB(x) (1−FA(x)) dx\Pr(p_B > p_A) = \int_0^1 F_A(x)\, f_B(x)\, dx, \qquad \mathbb{E}\bigl[\max(p_A - p_B,\, 0)\bigr] = \int_0^1 F_B(x)\,\bigl(1 - F_A(x)\bigr)\, dx

where FF and ff are the posterior CDF and density of each group. Probability-to-be-best across several variants and the credible interval on lift come from the same kind of integral. Nothing is simulated, so the same inputs always give exactly the same numbers.

A worked example

Suppose your control had 10,000 visitors and 1,000 conversions (10.0%), and your variant had 10,000 visitors and 1,120 conversions (11.2%). The pooled rate is 10.6%, giving a standard error of 0.435 percentage points and z=2.76z = 2.76. The two-tailed p-value is 0.006 — well below 0.05 — so the 12% relative lift is statistically significant at the 95% (and even 99%) confidence level. The 95% confidence interval on the relative lift runs from +3.0% to +21.0%: the true effect is very likely positive, but its size is still uncertain.

When to use it

  • After your test has reached its planned sample size (use the sample size calculator before launch).
  • For binary metrics: conversion, sign-up, click-through. For revenue or other continuous metrics, use the revenue-per-visitor calculator.
  • For 2+ variants: add variants and the calculator applies an omnibus chi-square test plus Holm-corrected pairwise comparisons automatically.

Common mistakes

  • Peeking. Checking daily and stopping when p first dips below 0.05 inflates false positives dramatically. Decide the sample size in advance, or use the sequential testing calculator, whose p-values remain valid under continuous monitoring.
  • One-tailed tests chosen after the fact. Halving the p-value by switching to one-tailed after seeing a positive result is p-hacking. Default to two-tailed.
  • Ignoring sample ratio mismatch (SRM). If your 50/50 test shows a badly uneven split of actual traffic, your assignment may be broken and every p-value in the test suspect. This calculator checks for SRM automatically; the standalone SRM checker is still available for checking splits on their own.
  • Multiple comparisons without correction. Each extra variant or metric multiplies your chances of a fluke. Use the built-in Holm correction for A/B/n tests.
  • Confusing significance with size. With enough traffic, a trivial +0.2% lift can be "significant". Read the confidence interval, not just the verdict.
What p-value counts as statistically significant?

By convention, a result is called statistically significant when the p-value is below 0.05, which corresponds to a 95% confidence level. This threshold is a trade-off, not a law of nature: at p < 0.05 you accept a 5% chance of declaring a winner when there is no real difference. For high-stakes or hard-to-reverse changes, many teams require p < 0.01 instead.

Should I use a one-tailed or two-tailed test?

Use a two-tailed test unless you have a specific, pre-registered reason not to. A two-tailed test can detect both improvements and regressions. A one-tailed test reaches significance faster, but it is blind to the possibility that your variant is worse — and switching to one-tailed after seeing the data is a form of p-hacking that inflates your false positive rate.

My test has been running for a week and p is 0.06. Can I keep it running until it hits 0.05?

Only if you planned the longer duration in advance. Repeatedly checking a fixed-horizon test and stopping the moment p dips below 0.05 (called "peeking") badly inflates the false positive rate — with daily checks, a nominal 5% error rate can become 20% or more. Either fix the sample size in advance with a sample size calculator, or use a sequential testing method with always-valid p-values.

Why does this calculator show both frequentist and Bayesian results?

Because they answer different questions about the same data. The frequentist panel answers: "if there were truly no difference, how surprising would this data be?" (p-values and confidence intervals). The Bayesian panel answers: "given this data, what is the probability the variant is better, and how much do I risk by shipping it?" (probability to be best, expected loss, and posterior distributions). Seeing both side by side is the fastest way to understand where the frameworks agree, where they diverge, and why. To compare like with like, the Bayesian probability is judged against a bar matched to your test: 97.5% for a two-tailed test at 95% confidence, 95% for a one-tailed one.

What is the SRM check that runs automatically?

SRM stands for sample ratio mismatch. Before you trust any p-value or posterior, the visitor counts you enter are compared against your expected traffic split (even by default, configurable under Advanced) with a chi-square test. If the observed split is far more uneven than random assignment can explain (p < 0.001), the calculator warns you that the test data itself is suspect — a broken split invalidates every metric downstream.

What happens when I test more than one variant (A/B/n)?

Each extra variant adds another comparison, and every comparison carries its own false positive risk. Testing 4 variants at α = 0.05 gives up to an 18.5% chance of at least one false positive if you do not correct for it. This calculator runs an omnibus chi-square test across all groups and adjusts pairwise p-values with the Holm (default) or Bonferroni correction to keep the overall error rate at your chosen α.

Do visitors and conversions need to be unique counts?

Yes — the test assumes each visitor is an independent trial that either converted or did not. Use unique visitors (or users, or sessions — consistently) and count each converter once. Counting total conversion events, where one user can convert repeatedly, violates the independence assumption and makes the p-value unreliable.

  1. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value does and does not tell you.
  2. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70.The step-down correction applied to A/B/n comparisons.
  3. Campbell, I. (2007). Chi-squared and Fisher–Irwin Tests of Two-by-Two Tables with Small Sample Recommendations. Statistics in Medicine, 26(19), 3661–3675.Choosing between the chi-squared and exact tests for small samples.
  4. Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association, 22(158), 209–212.The score interval drawn around each conversion rate.
  5. Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science, 16(2), 101–133.Why the Wilson interval is preferred to the simple Wald interval.
  6. Deng, A., Knoblich, U., & Lu, J. (2018). Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’18); arXiv:1803.06336.The delta method behind the interval on relative lift.
  7. Casella, G., & Berger, R. L. (1987). Reconciling Bayesian and Frequentist Evidence in the One-Sided Testing Problem. Journal of the American Statistical Association, 82(397), 106–111.Why a one-sided p-value and a Bayesian probability line up.
  8. Stucchio, C. (2015). Bayesian A/B Testing at VWO. Visual Website Optimizer technical whitepaper.Expected loss as a Bayesian decision rule.
  9. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.The sample ratio mismatch check and what causes a mismatch.

Every calculation here is tested against an independent source. How the numbers are checked