What this calculator does
Paste the p-values of everything you looked at in a test: every metric, every segment, every variant against control. The calculator shows which results are still significant once you account for how many there were, under two standard corrections, and gives the adjusted p-value for each.
The two corrections
Holm sorts the p-values from smallest to largest and compares the smallest with α / m, the next with α / (m − 1), and so on, stopping at the first that fails. It controls the chance of any false positive at all.
Benjamini-Hochberg also sorts them, and finds the largest rank k whose p-value is at most k α / m; that result and every smaller one survive. It controls the false discovery rate: of the results you keep, the expected share that are false.
A worked example
A test reports twelve metrics, with p-values from 0.001 to 0.251. Taken one at a time, 5 are below 0.05. After Holm's correction 1 remains; after Benjamini-Hochberg, 2. The three that drop out entirely, with p-values around 0.04, are exactly the kind of result twelve looks produce by chance.
Good habits
- Name one primary metric before the test. It needs no correction and carries the decision.
- Treat corrected secondary results as leads. A segment that survives is worth a follow-up test, not a launch.
- Count every look. Metrics, segments and variants all multiply. For several variants against one control, the A/B test calculator applies the correction for you.
- Remember looks over time count too. Checking one metric repeatedly is its own multiple comparisons problem: see the peeking penalty calculator.