Skip to content
TESTING HAVEN

IV · LEARN

The Pocket Ledger

Fifteen checks for a test you can trust, on one printed page

Most bad A/B test results come from a handful of avoidable mistakes. These 15 checks cover them, in the order they come up. Tick them off here, or print the page and keep it by your desk: it fits on one sheet.

Before you start

  1. Choosing the metric after the test is choosing the answer.

    What a primary metric is →

  2. It is a business decision, not a forecast.

    MDE & duration calculator →

  3. A number fixed in advance cannot be argued with later.

    Sample size calculator →

  4. Weekday and weekend visitors behave differently.

    MDE & duration calculator →

  5. Sequential testing calculator →

  6. What a guardrail metric is →

While it runs

  1. A broken split is cheapest to find early.

    SRM checker →

  2. Stop at the first of 5 looks that crosses the line, and 14% of changes that do nothing become winners.

    Peeking penalty calculator →

  3. A test that changed halfway is two half-tests.

When it ends

  1. If the split is wrong, no other number can be trusted.

    SRM checker →

  2. A/B test calculator →

  3. Significant at 95% does not mean 95% likely to be real.

    False positive risk calculator →

  4. Winning tests overstate their lift.

    Winner’s curse calculator →

  5. Too many metrics calculator →

  6. Tests that lost or showed nothing are part of the record.

    A/B test report →

How to use it

The list is short on purpose. Each line is a decision that is easy to make before a test and almost impossible to make honestly afterwards: which metric counts, how many visitors you need, when you are allowed to look. Settle those first and most of the statistics takes care of itself.

If you only adopt three, make them these: fix the sample size before you start, do not stop early because the result looks good, and check the traffic split before you read anything else. The A/B test report runs the checks in the last section for you, and the glossary explains any term here that is new.

  1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.The standard practitioner’s reference for running tests that can be trusted.
  2. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.Why the traffic split is checked before anything else.
  3. Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated Significance Tests on Accumulating Data. Journal of the Royal Statistical Society: Series A (General), 132(2), 235–244.What stopping at the first significant look does to the false positive rate.
  4. Kohavi, R., Deng, A., & Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22).Why significant at 95% does not mean 95% likely to be real.
  5. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science, 9(6), 641–651.Why winning tests overstate their lift.