What this calculator does
It reads a whole testing program at once. From a list of past tests it estimates how many of your wins are real, what they add up to once the luck is taken out, and how large real lifts typically are in your program. That last figure is the one the winner's curse calculator and the test report ask you to supply, so this page turns a guess into an estimate from your own record.
How it works
Each test has an observed lift and a standard error , recovered from its p-value. The true lifts are treated as varying around a program average with standard deviation . The spread is estimated from how much more the observed lifts vary than their standard errors can explain. Each test's expected true lift is then
a noisy test (large ) is pulled most of the way to the average, and a precise one keeps most of its own estimate. The estimate of is the standard one from meta-analysis, due to DerSimonian and Laird.
A worked example
The example has 16 tests. 4 reached significance, with observed lifts that compound to +55.9%. Read together, the record suggests most real lifts in this program fall between -6.4% and +10.3%. After shrinkage the same 4 wins are expected to deliver +26.2%, about 47% of the claim, and about 3.9 of them are expected to be real improvements at all.
What it assumes
- You entered every test. A list of winners only cannot be corrected, because the information about luck is in the tests that were left out.
- Lifts are relative and the p-values two-tailed. That is what most testing tools report.
- The wins are independent launches. The totals compound the lifts as if each win were shipped on top of the last and affected the same metric.
- Real lifts are roughly bell-shaped around an average. A program with a few enormous wins among many duds fits this less well, and its big wins will be shrunk somewhat too much.