Skip to content
TESTING HAVEN

III · DIAGNOSE

False Positive Risk Calculator

Is your significant result real?

False positive risk is the chance that a statistically significant win is not a real improvement. If one in ten of the ideas you test really works, a win at 95% confidence and 80% power has a 22.0% chance of being a false positive, not 5%.

Your testing program

%
%
Confidence
Test type

A statistically significant win from a test like this has a 22.0% chance of being a false positive: about 1 in 5 such wins is not a real improvement. The chance the win is real is 78.0%.

This is a different number from the 5% behind 95% confidence. Confidence limits how often a change with no effect looks like a winner (2.5% of the time here, because only results in the variant’s favour are shipped). How many of your winners are real also depends on how many of your ideas work and how reliably your tests detect the ones that do.

Out of 1,000 tests like this

IdeasTestsSignificant winNo significant win
Really work10080 real wins20 missed
Have no effect90022.5 false wins877.5

Of the 102.5 significant wins, 22.5 come from ideas with no effect: 22.0%.

How to lower the risk

  • Repeat the test. If an independent repeat also comes out as a significant win, the risk that the result is a false positive falls to 0.9%.
  • Use a stricter threshold. For a risk of 5% at this success rate and power, a win needs p < 0.0094 (two-tailed). A stricter threshold needs more visitors; the sample size calculator shows how many.
  • Keep power high. With these settings the risk would be 31.0% at 50% power and 20.0% at 90% power. Underpowered tests find fewer real wins, so false ones make up more of what gets shipped.

Risk by success rate

The rust-colored dot marks your inputs. The fewer of your ideas work, the more of your significant wins are false.

Reported byIdeas that workFalse positive risk
Airbnb Search8%26.4%
Booking.com, Google Ads, Netflix10%22.0%
Bing15%15.0%
Avinash Kaushik20%11.1%
Microsoft33%5.9%

Success rates as reported in Kohavi, Deng and Vermeer (2022), listed under Sources below. The risk in each row is recalculated for your confidence level, test type and power.

How the mathematics works

What this calculator does

This calculator answers the question that follows every winning A/B test: how likely is it that this win is real? Enter the share of your tested ideas that really work, the confidence level and power of your tests, and it returns the false positive risk: the probability that a statistically significant win comes from a change with no effect at all.

The answer is usually larger than people expect. If one idea in ten works, a significant win at 95% confidence and 80% power has a 22.0% chance of being a false positive, not 5%.

To get a feel for it, play Real or noise?: ten tests that all reached significance, and you guess which wins are real.

Why it is not 5%

A 95% confidence level means that a change with no effect will look significant 5% of the time. That is a statement about changes with no effect. It says nothing about how many such changes you test. If most of your ideas do nothing, most of your tests are chances for a false win, and the few real wins have to share the stage with them.

A medical screening test works the same way. A test that is wrong for only 5% of healthy people still produces mostly false alarms if the condition is rare, because there are so many healthy people to be wrong about.

The formula

Let π\pi be the share of tested ideas that really work, 1−β1 - \beta the power, and αw\alpha_w the chance that an idea with no effect shows a significant win. Bayes' rule gives:

risk=P(no effect∣significant win)=αw (1−π)αw (1−π)+(1−β) π\text{risk} = P(\text{no effect} \mid \text{significant win}) = \frac{\alpha_w\,(1 - \pi)}{\alpha_w\,(1 - \pi) + (1 - \beta)\,\pi}

For a two-tailed test, αw=α/2\alpha_w = \alpha / 2: a no-effect change is equally likely to look significantly better or significantly worse, and only the first is shipped as a win. For a one-tailed test, αw=α\alpha_w = \alpha.

If an independent repeat of the test also wins, both terms are squared: αw2(1−π)\alpha_w^2 (1 - \pi) against (1−β)2π(1 - \beta)^2 \pi. Two false wins in a row are far less likely than two real ones, which is why replication is so effective.

A worked example

Suppose 10% of the ideas you test really improve the metric, and you run two-tailed tests at 95% confidence with 80% power. Out of 1,000 tests, 100 are of ideas that work, and 80 of those are detected. The other 900 are of ideas with no effect, and 2.5% of them, 22.5 tests, come out as significant wins anyway. So of 102.5 significant wins, 22.5 are false: 22.0%. The chance a given win is real is 78.0%.

If you repeat a winning test and it wins again, the risk falls to 0.9%. To reach a 5% risk from a single test instead, you would need p < 0.0094, which takes more visitors (see the sample size calculator). A program where one idea in three works starts from a much better place: 5.9%.

Finding your success rate

The success rate is the one input you have to estimate. The best source is your own record: the share of finished tests that produced a significant win on the metric they were designed to move. That observed share contains some false wins and misses some real ones, and the two roughly cancel. If ww is your observed share of wins, the implied share of ideas that work is π=(w−αw)/(1−β−αw)\pi = (w - \alpha_w) / (1 - \beta - \alpha_w). With 12% of tests winning, that gives about 12%.

Published figures from large experimentation programs run from about 8% to 33%, and the calculator shows the risk at each of those. If you have no record yet, treat the result as a range rather than a number.

What it assumes

  • Two kinds of idea. Each idea either has no effect or improves the metric by the amount the test was powered for. Real effects come in all sizes; smaller ones are detected less often, which raises the risk. Ideas that make things worse are counted as having no effect, which slightly overstates it.
  • A clean test. The sample size was fixed in advance, nobody stopped early on a good day, there is one primary metric, and the traffic split is healthy. Peeking (see the sequential testing calculator), several metrics, and sample ratio mismatch (see the SRM checker) all push the real false positive rate above the nominal one.
  • Your estimate of the success rate. The result is only as good as that input, so try more than one value.

How this differs from the Bayesian reading

The A/B test calculator reports a Bayesian probability that the variant beats control. By default that uses a flat prior, which treats every conversion rate as equally likely before the data arrive and takes no account of your track record. False positive risk does the opposite: it ignores the size of the observed difference and uses only the fact that the result was significant, together with how often your ideas work. The two answer different questions, and a result is on firmest ground when both are reassuring.

Common mistakes

  • Reading 95% confidence as a 95% chance the variant is better. Confidence describes how the test behaves when nothing is going on, not how likely your result is to be true.
  • Reading the p-value as the false positive risk. A p-value of 0.03 does not mean a 3% chance the win is false.
  • Trusting wins from underpowered tests. Low power leaves false wins as a larger share of what comes out significant (check yours with the power calculator).
  • Shipping a surprising, high-stakes win without repeating it. A second independent win is the cheapest way to cut the risk.
What is false positive risk?

False positive risk is the chance that a statistically significant win is not a real improvement. It answers the question people usually think the p-value answers: "given that my test came out significant, how likely is it that the change does nothing?" With 10% of ideas working, a two-tailed test at 95% confidence and 80% power, it is 22.0%.

Is the false positive risk not just 5%?

No. The 5% (the significance level) is the chance of a significant result when the change has no effect. False positive risk runs the other way: the chance the change has no effect when the result is significant. The two are different quantities, and the second depends on how many of the ideas you test really work. If few ideas work, most tests are of changes with no effect, so even a small false positive rate produces many false wins next to the few real ones.

How do I find my success rate?

Look at your own history: the share of finished tests that produced a significant win for the metric they were meant to improve. That share includes some false wins and misses some real ones, but the two roughly offset. For example, if 12% of your tests were significant wins, the implied share of ideas that really work is about 12%. Reported rates in large experimentation programs run from about 8% to 33%. If you have no history, try 10% and 30% to see the range.

Why does low power raise the risk?

An underpowered test finds fewer of the real improvements, while changes with no effect produce false wins at the same rate as before. So false wins make up a larger share of everything that comes out significant. With 10% of ideas working, dropping from 80% power to 30% raises the false positive risk from 22.0% to 43%.

Should I use p < 0.01 instead of p < 0.05?

It depends on how many of your ideas work and what a wrong decision costs. With 10% of ideas working and 80% power, a 5% false positive risk needs p < 0.0094 in a two-tailed test, which is close to 0.01. A stricter threshold needs more visitors for the same power. Repeating the test is the other route: if an independent repeat also wins, the risk falls to 0.9%.

What does this calculator assume?

It treats every tested idea as one of two kinds: no effect, or an improvement of the size the test was powered to detect. It also assumes the test was run cleanly: a sample size fixed in advance, no stopping early on a good-looking result, one primary metric, and a healthy traffic split. Peeking, several metrics and sample ratio mismatch all push the real false positive rate above the nominal level, and the risk with it.

  1. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).The false positive risk formula, and the success rates reported by large experimentation programs.
  2. Ioannidis, J. P. A. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2(8), e124.The original argument that the share of true findings depends on how many tested ideas are true, and on power.
  3. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., et al. (2018). Redefine Statistical Significance. Nature Human Behaviour, 2(1), 6–10.The case for a stricter threshold when few tested ideas work.
  4. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.What a p-value does and does not tell you.

Every calculation here is tested against an independent source. How the numbers are checked