IV · LEARN
How the Numbers Are Checked
Every statistic here, tested against an independent source
Every statistic on Testing Haven is compared with an answer produced somewhere else: by SciPy or statsmodels, the standard scientific libraries for Python, or by a table printed in a published paper. The table below is rebuilt from the live code each time the site is published. It currently covers 24 checks, 135 reference cases and 403 individual numbers.
The accuracy table
| What is checked | Compared against | Cases | Largest difference found |
|---|---|---|---|
| Significance of a difference between two conversion ratesTwo-proportion z-test | Compared against: statsmodels (proportions_ztest) | Reference cases: 6 | Agrees to 14 significant digits |
| Significance when conversions are fewFisher’s exact test | Compared against: SciPy (fisher_exact) | Reference cases: 5 | Agrees to 11 significant digits |
| Confidence interval for a conversion rateWilson score interval | Compared against: statsmodels (proportion_confint) | Reference cases: 4 | Identical to 15 digits |
| Sample ratio mismatch checkChi-square goodness-of-fit test | Compared against: SciPy (chisquare) | Reference cases: 4 | Agrees to 12 significant digits |
| Overall test across several variantsChi-square test of independence | Compared against: SciPy (chi2_contingency) | Reference cases: 3 | Agrees to 13 significant digits |
| Correction for several comparisonsHolm step-down adjustment | Compared against: statsmodels (multipletests) | Reference cases: 3 | Identical to 15 digits |
| False discovery rate across many metricsBenjamini–Hochberg adjustment | Compared against: statsmodels (multipletests) | Reference cases: 6 | Identical to 15 digits |
| Difference in revenue per visitorWelch’s t-test, with its confidence interval | Compared against: SciPy (ttest_ind) | Reference cases: 3 | Agrees to 7 significant digits |
| Rank-based check for skewed revenueMann–Whitney U test | Compared against: SciPy (mannwhitneyu) | Reference cases: 3 | Agrees to 13 significant digits |
| The beta distribution behind every Bayesian resultBeta cumulative distribution | Compared against: SciPy (beta.cdf) | Reference cases: 11 | Agrees to 9 decimal places |
| Probability the variant beats the controlNumerical integration over both posteriors | Compared against: SciPy numerical integration | Reference cases: 9 | Agrees to 9 decimal places |
| Expected loss from shipping the variantNumerical integration over both posteriors | Compared against: SciPy numerical integration | Reference cases: 9 | Agrees to 8 significant digits |
| Probability to be best among several variantsNumerical integration over every posterior | Compared against: SciPy numerical integration | Reference cases: 2 | Agrees to 10 decimal places |
| Credible interval for the relative liftNumerical integration and root finding | Compared against: SciPy numerical integration | Reference cases: 3 | Agrees to 9 decimal places |
| How much a significant result overstates the truthPower, sign error and exaggeration ratio | Compared against: SciPy numerical integration | Reference cases: 8 | Agrees to 11 significant digits |
| Expected true lift of a winning testShrinkage toward zero with a normal prior | Compared against: SciPy numerical integration | Reference cases: 6 | Identical to 15 digits |
| False positive rate when a test is checked repeatedlyRecursive numerical integration | Compared against: SciPy (joint normal probability) | Reference cases: 10 | Agrees to 5 decimal places |
| The same, against the original published tableFalse positive rate for 1 to 100 looks | Compared against: Armitage, McPherson & Rowe (1969), printed table | Reference cases: 9 | Every printed value reproduced |
| Chance that a significant result is a false positiveBayes’ rule over ideas that work and ideas that do not | Compared against: Kohavi, Deng & Vermeer (2022), printed table | Reference cases: 5 | Every printed value reproduced |
| Typical effect across a program of testsDerSimonian–Laird random-effects model | Compared against: statsmodels (combine_effects) | Reference cases: 4 | Agrees to 14 significant digits |
| The revenue-like distribution on the distributions pageTweedie density and chance of zero | Compared against: statsmodels (Tweedie family) | Reference cases: 7 | Agrees to 9 significant digits |
| Statistical power of a planned testNormal approximation | Compared against: Exact power, by adding up every possible outcome | Reference cases: 5 | Within 0.2 percentage points |
| Sample size for a conversion-rate testNormal approximation with pooled variance | Compared against: statsmodels (NormalIndPower), a different formula | Reference cases: 3 | Within 0.20% of the sample size |
| Sample size for a revenue testNormal approximation for a difference in means | Compared against: statsmodels (NormalIndPower) | Reference cases: 7 | Within 1 visitor |
Reference values were produced with SciPy 1.18.1 and statsmodels 0.15.0. A case is one set of inputs; most cases compare several numbers, such as a test statistic, a p-value and both ends of an interval.
How to read the table
Most rows compare two ways of computing exactly the same quantity, and they agree to many digits. “Agrees to 9 significant digits” means the first nine digits of the site’s answer and the reference answer are the same, in the worst case found. For a p-value of 0.0312345678, any difference would be beyond the last digit shown here.
Three rows are different in kind. Power and the two sample size rows compare the planning formula with a different method: the exact power found by adding up every possible outcome, or a library that uses another standard formula. Planning formulas are approximations, so small differences are expected. The row shows how small: a fraction of a percentage point of power, a fraction of a percent of the sample size, or a single visitor.
One row is limited by its reference. For the peeking calculation, SciPy’s own answer is only accurate to about five decimal places, so that is as far as the comparison can go. The same calculation also reproduces every value in the original 1969 table.
The rule behind it
A number worked out from the formula being tested is not a check. If the formula is wrong, the check is wrong in the same way and passes anyway. So every reference value here comes from separate software, written by other people, or from a table printed in a paper, typed in at the precision the paper printed.
The worked examples in the articles follow the same idea from the other side: the figures in the text are computed by the calculators’ own code when the site is built, so the words cannot drift away from what the calculator shows.
What each calculator uses
- A/B test calculator
- Two-proportion z-test, switching to Fisher’s exact test when conversions are few. Wilson intervals for each rate. Holm or Bonferroni correction across variants. Bayesian results from beta posteriors by numerical integration, not simulation, so the same inputs always give the same answer.
- A/B test report
- The same significance test, plus a traffic-split check, the power of the test for the lift you were looking for, and a shrunk estimate of the true lift.
- Sample size, power and MDE calculators
- The normal approximation for two proportions with pooled variance. With several variants, alpha is divided among the comparisons with the control.
- Sequential testing calculator
- An always-valid p-value from the mixture sequential probability ratio test, with a normal prior on the effect.
- Revenue per visitor calculator
- Welch’s t-test with its confidence interval, and the Mann–Whitney U test as a rank-based check.
- Sample size for revenue
- The normal approximation for a difference in means, from the standard deviation of revenue per visitor.
- CUPED calculator
- The variance left after adjusting for a pre-test measure: one minus the squared correlation.
- False positive risk calculator
- Bayes’ rule over ideas that work and ideas that do not, given alpha, power and your success rate.
- Winner’s curse calculator
- Shrinkage toward zero under a normal prior, and the sign-error and exaggeration calculations of Gelman and Carlin.
- Peeking penalty calculator
- The chance that any of several evenly spaced looks crosses the line, by recursive numerical integration, and the constant threshold that brings it back to alpha.
- Too many metrics calculator
- Bonferroni and Holm corrections for the family-wise error rate, and the Benjamini–Hochberg procedure for the false discovery rate.
- Program health check
- A random-effects model of your past tests to estimate how large real effects usually are, then shrinkage of each result toward that.
- SRM checker
- A chi-square goodness-of-fit test of the visitor counts.
What the calculators assume
- Each visitor counts once and independently. If one person can appear many times, or visitors influence each other, the real uncertainty is larger than any calculator will show.
- The test was read once, at a size fixed in advance. Ordinary p-values and intervals assume that. If you looked early and stopped on a good result, use the peeking penalty or the sequential calculator.
- The counts are right. A calculator cannot see a tracking fault. The traffic-split check catches one common kind; nothing catches them all.
- Planning numbers are approximations. Sample sizes and durations are estimates to plan around, not guarantees.
Your numbers stay with you
The calculators run in your browser: nothing you type is sent anywhere to be worked out. Your inputs are kept in the page address, so a result can be bookmarked or shared as a link. The site counts page visits and a few actions, such as “a calculation was completed”, and that counter is given the page name only, never the address with your numbers in it, and it uses no cookies. Opening a shared link works like opening any other link: the address is what your browser asks the web server for.
Kept in order automatically
Every change to the site runs the reference checks above, a browser test of each calculator, and an accessibility scan of every page on a desktop and a phone-sized screen. Once a week the published sources cited across the site are checked to make sure each link still leads to the paper it claims to.
Who publishes it
Testing Haven is published by camploanalytics LLC. The calculators are free and need no account. New terms are explained in the glossary, and the checklist puts the whole method on one page.
Sources
- Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244.The printed table the peeking calculation is checked against.
- Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).The printed table the false positive risk calculation is checked against.
- Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.The sign-error and exaggeration calculations.