Statistical Significance in A/B Testing, Explained
What p-values, confidence levels, and significance actually mean in an A/B test — and the mistakes that make "significant" results wrong.
Companion tool: The A/B Test Calculator
IV · LEARN
The library
Essays on the statistics of experimentation, written for people who run tests.
01Statistical significance, explained02How to size an A/B test03Bayesian vs frequentist04The peeking problem05Sample ratio mismatch
How to Size an A/B Test (Sample Size, MDE, and Power)
A practical walkthrough of sample size calculations: choosing an MDE, setting power, and translating the answer into test duration.
Companion tool: Sample Size
Bayesian vs Frequentist A/B Testing: Which Should You Use?
How Bayesian probability-to-be-best and expected loss compare to p-values and confidence intervals, and when each approach fits.
Companion tool: Bayesian
The Peeking Problem: Why Checking Results Early Breaks Your Test
Why repeatedly checking a fixed-horizon test inflates false positives, with simulations, and how sequential methods like mSPRT fix it.
Companion tool: Sequential
Sample Ratio Mismatch (SRM): The Silent A/B Test Killer
How to detect sample ratio mismatch, the most common causes, and why you should throw away results from a test that fails an SRM check.
Companion tool: SRM Checker