Testing Haven

A/B Test Sample Size Calculator

How many visitors does your test need?

Test parameters

%
%
MDE type(a 5% lift on 10% → 10.50%)
%
Test type(two-tailed is the safer default)

You need 57,763 visitors per variant (115,526 total for a 50/50 A/B test) to detect a change from 10.00% to 10.50% with 95% confidence and 80% power. With fewer visitors, a real effect of this size would go undetected more than 20% of the time. To translate this into a runtime, use the test duration calculator.

Sample size vs. minimum detectable effect

Halving the MDE roughly quadruples the required sample size (note the log scale). The pink dot marks your current inputs.

What this calculator does

This sample size calculator tells you how many visitors each variant of your A/B test needs before you start. You give it four numbers — your baseline conversion rate, the minimum detectable effect (MDE) you care about, your significance level, and your desired statistical power — and it returns the required sample size per variant, with a chart showing how that requirement explodes as the MDE shrinks.

Sizing the test up front is the single highest-leverage habit in experimentation: it prevents both the underpowered test that wastes four weeks to conclude nothing, and the endless test that gets stopped the day it flickers significant.

The formula

For a two-proportion test comparing baseline rate p1p_1 against target rate p2p_2 (baseline + MDE), the required sample size per variant is:

n=(z1−α/22pˉ(1−pˉ)+z1−βp1(1−p1)+p2(1−p2))2(p2−p1)2n = \frac{\left( z_{1-\alpha/2} \sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta} \sqrt{p_1(1-p_1) + p_2(1-p_2)} \right)^2}{(p_2 - p_1)^2}

where pˉ=(p1+p2)/2\bar{p} = (p_1 + p_2)/2 is the average rate, z1−α/2z_{1-\alpha/2} is the normal quantile for your significance level (1.96 at 95%, two-tailed), and z1−βz_{1-\beta} is the quantile for your power (0.84 at 80%). The denominator is the squared effect — which is why halving the MDE roughly quadruples the required sample.

A worked example

Baseline conversion rate 10%, and you want to detect a 5% relative lift (10% → 10.5%) at 95% significance and 80% power, two-tailed. Plugging in: p1=0.10p_1 = 0.10, p2=0.105p_2 = 0.105, pˉ=0.1025\bar{p} = 0.1025. The formula gives 57,763 visitors per variant, or about 115,500 total. If your site gets 20,000 visitors a week into the test, that is a six-week runtime — knowledge worth having before you build the variant, not after. (Check the MDE & duration calculator to see what is detectable at your actual traffic.)

How to choose each input

  • Baseline rate: pull the last 2–4 weeks of the exact metric and audience your test will use. A stale or mismeasured baseline skews everything.
  • MDE: work backwards from business value — the smallest lift that pays for the engineering time. Anything smaller than that is not worth detecting.
  • Significance: 95% is standard; use 99% for risky, hard-to-reverse changes.
  • Power: 80% is standard; 90% for decisions where missing a real winner is costly.

Common mistakes

  • Stopping early at the first significant reading. The sample size is the contract that makes your error rates real. Breaking it by peeking voids the warranty — use the sequential testing calculator if you need early stopping.
  • Relative/absolute MDE confusion. A "5% lift" on a 10% baseline could mean 10.5% or 15% depending on interpretation, with a ~9x difference in required sample. This calculator labels both explicitly.
  • Sizing for the hoped-for effect instead of the minimum worthwhile effect. Optimism makes tests small and inconclusive.
  • Forgetting that per-variant means per variant. An A/B/C test needs the quoted n in each of the three arms.
  • Not running full weeks. Round your runtime up to whole weeks so weekday/weekend mix does not bias the sample.

Frequently Asked Questions

Why do I need to calculate sample size before starting a test?

Because the alternative is running until the result "looks done", which is how false positives are manufactured. Fixing the sample size in advance guarantees your stated error rates actually hold: your significance level controls false positives, and your power controls false negatives. It also tells you before you invest whether the test is feasible at all at your traffic level.

What is a reasonable minimum detectable effect (MDE)?

The MDE should be the smallest lift that would still be worth shipping — not the lift you hope to get. For most mature conversion funnels, real UI changes move metrics by 1–10% relative, so MDEs in the 2–10% range are typical. Picking an unrealistically large MDE (like 25%) gives you a small, fast test that will almost certainly end "not significant" because real effects that big are rare.

What do 80% power and 95% significance actually mean?

95% significance (alpha = 0.05) means that if the variant truly changes nothing, you have at most a 5% chance of a false alarm. 80% power (beta = 0.2) means that if the true effect is exactly your MDE, you have an 80% chance of detecting it — and a 20% chance of missing it. These are the conventional defaults; raise power to 90% for important decisions if you can afford the roughly 34% larger sample.

What is the difference between relative and absolute MDE?

A relative MDE is a percentage of your baseline: a 10% relative lift on a 5% conversion rate means moving to 5.5%. An absolute MDE is in percentage points: a 1-point absolute lift on 5% means moving to 6%. Confusing the two changes sample sizes enormously — a 1-point absolute lift on a 5% baseline is a 20% relative change. This calculator supports both; the toggle shows you the implied target rate either way.

Can I stop the test early if it reaches significance before the planned sample size?

Not with a fixed-horizon test — stopping at the first significant reading is peeking and inflates your false positive rate well beyond 5%. If you want the option to stop early, use a sequential testing method (see our sequential testing calculator), which charges a slightly larger sample in exchange for always-valid results.

Does this calculator match Evan Miller and statsmodels?

Yes. It uses the standard two-proportion power formula with exact normal quantiles and no continuity correction, the same approach as Evan Miller's calculator, and its outputs are unit-tested against reference values. Small differences versus other tools usually come down to the continuity correction or one- vs two-tailed defaults.