Skip to content
TESTING HAVEN

III · DIAGNOSE

Multiple Comparisons Calculator

Too many metrics? See which results survive

Look at enough metrics and something will reach significance by luck alone. With 12 metrics and nothing real, the chance that at least one shows p < 0.05 is 46%. Paste your p-values to see which survive a correction: in the example on this page, 5 look significant, 2 survive a false discovery rate correction and 1 survives a strict one.

Your results

Confidence

Paste the p-values of everything you looked at, not only the ones that reached significance.

How the mathematics works

What this calculator does

Paste the p-values of everything you looked at in a test: every metric, every segment, every variant against control. The calculator shows which results are still significant once you account for how many there were, under two standard corrections, and gives the adjusted p-value for each.

The two corrections

Holm sorts the p-values from smallest to largest and compares the smallest with α / m, the next with α / (m − 1), and so on, stopping at the first that fails. It controls the chance of any false positive at all.

Benjamini-Hochberg also sorts them, and finds the largest rank k whose p-value is at most k α / m; that result and every smaller one survive. It controls the false discovery rate: of the results you keep, the expected share that are false.

A worked example

A test reports twelve metrics, with p-values from 0.001 to 0.251. Taken one at a time, 5 are below 0.05. After Holm's correction 1 remains; after Benjamini-Hochberg, 2. The three that drop out entirely, with p-values around 0.04, are exactly the kind of result twelve looks produce by chance.

Good habits

  • Name one primary metric before the test. It needs no correction and carries the decision.
  • Treat corrected secondary results as leads. A segment that survives is worth a follow-up test, not a launch.
  • Count every look. Metrics, segments and variants all multiply. For several variants against one control, the A/B test calculator applies the correction for you.
  • Remember looks over time count too. Checking one metric repeatedly is its own multiple comparisons problem: see the peeking penalty calculator.
What is the multiple comparisons problem?

Every significance test has a 5% chance of a false positive when nothing is going on. Run one test and that is the whole risk. Check 5 metrics and the chance that at least one is falsely significant is 23%; check 20 and it is 64%. The more places you look, the more likely it is that something will look like a discovery by luck.

Which results do I need to include?

Everything you looked at, not only what turned out significant. The correction depends on how many chances chance had. If you examined twelve metrics and paste only the five interesting ones, the correction will be far too lenient. The same applies to segments: "it worked for mobile users in Germany" is one of many slices you could have found.

What is the difference between Holm and Benjamini-Hochberg?

They protect against different things. Holm keeps the chance of even one false positive, anywhere in the list, at your chosen level. Benjamini-Hochberg lets a few through and keeps the expected share of false positives among the survivors at that level. Holm is the right choice when a single wrong claim is costly; Benjamini-Hochberg when you are screening many results for leads to follow up.

Do I need to correct my primary metric?

Not if there is exactly one, chosen before the test. A single pre-registered primary metric is one comparison and needs no correction. The correction is for everything else: secondary metrics, segments, and extra variants. That is why naming the primary metric in advance is so valuable.

Why not use Bonferroni?

Bonferroni multiplies every p-value by the number of comparisons. Holm gives the same guarantee and is never stricter, because it relaxes the multiplier for each result after the smallest. There is no situation in which Bonferroni keeps a result that Holm rejects, so Holm is the better default.

  1. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70.The step-down correction that controls the chance of any false positive.
  2. Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.The false discovery rate and the procedure that controls it.
  3. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).How looking at many metrics and segments produces false discoveries in A/B tests.

Every calculation here is tested against an independent source. How the numbers are checked