What this calculator does
This statistical power calculator answers the question every "not significant" test result should trigger: could this test even have detected the effect I care about? Enter your baseline conversion rate, the effect size that would matter to your business, and your per-variant sample size, and it returns the power of your test — the probability of detecting that effect if it is real — plus the smallest lift your sample can reliably detect at 80% power.
Use it before launch to sanity-check a planned test, or after a null result to decide whether "no significant difference" means "no effect" or just "not enough data".
The formula
For a two-proportion test with per-variant sample size , baseline rate , and target rate , the power of a two-tailed test at significance level is:
where is the average of the two rates, is the standard normal CDF, and is the critical value (1.96 at 95% confidence). Intuitively: power is the chance that your observed difference clears the significance bar, computed under the assumption that the true effect equals your MDE.
With more than one variant, is divided by the number of variants (a Bonferroni adjustment), matching the multiple-comparison correction applied when the test is analyzed. Extra variants therefore lower the power of each comparison at a given sample size.
A worked example
You ran a test with 10,000 visitors per variant on a 10% baseline, hoping for a 5% relative lift (10% → 10.5%). Plugging in gives power of roughly 21% — meaning that even if the variant truly delivered the hoped-for lift, this test would detect it only about one time in five. A null result from this test is nearly meaningless. To reach 80% power for that effect you would need about 57,800 visitors per variant (see the sample size calculator), or you could re-scope the test toward a bigger, more detectable change.
When to use it
- Before launch: verify the planned sample size gives ≥80% power for the minimum worthwhile effect.
- After a null result: distinguish "well-powered null — the effect probably is not there" from "underpowered null — we learned little".
- When auditing someone else's readout: underpowering is the most common silent flaw in test claims.
Common mistakes
- Computing post-hoc power from the observed effect. It is a disguised p-value and adds no information; always evaluate power against the effect size that matters, not the one you happened to observe.
- Treating a null result from a 20%-power test as evidence of no effect. Absence of evidence is only evidence of absence when power is high.
- Ignoring the winner's curse. Significant results from underpowered tests systematically overstate the true lift — budget for shrinkage when projecting impact.
- Quietly relaxing alpha or switching to one-tailed after the fact to rescue an underpowered test — decide rigor settings before launch.