Skip to content
TESTING HAVEN

IV · LEARN

Sample Ratio Mismatch (SRM): The Silent A/B Test Killer

By Testing Haven · Published · Updated

Sample ratio mismatch (SRM) is the silent killer of A/B tests: a traffic split that deviates from its configuration by more than chance allows, revealing that users were sorted into groups by some systematic force rather than the coin flip your statistics assume. This guide covers detection, causes, and the uncomfortable question of what to do with an affected test.

Why a small imbalance is a big deal

The threat is not the imbalance itself — tests tolerate unequal group sizes fine. The threat is what the imbalance proves: some mechanism is removing or adding users non-randomly. Missing users are never a random sample (they are the slow connections, the bot-filtered, the crash victims), so the groups no longer differ only by treatment. A bias of a percent or two can produce metric deltas as large as the treatment effects you are trying to measure — in either direction.

Here is how that plays out. Take a 50/50 test with 500,000 users assigned to each arm and no treatment effect at all: both arms convert at 10%. Now suppose the variant's redirect loses 2% of its users before they are logged, and that the lost users are slow-connection visitors who would not have converted anyway. The variant shows 50,000 conversions from 490,000 users: 10.2% against 10.0%, a +2.0% "lift" with p = 0.0008. A clean, highly significant win, produced entirely by missing data. The only thing standing between that result and a launch announcement is the SRM check, which flags the 500,000 / 490,000 split with χ² = 101 and a p-value around 10⁻²³.

Detection

A chi-square goodness-of-fit test compares observed counts to the configured split; the SRM checker does it in seconds with a plain-English verdict. For each group, take the observed count minus the expected count, square it, and divide by the expected count. Add those up and compare the total with a chi-square distribution with one fewer degrees of freedom than there are groups.

A worked example: you configured 50/50 and logged 50,600 control users against 49,400 variant users. Expected is 50,000 each, so χ² = 600²/50,000 + 600²/50,000 = 14.4 and p = 0.00015. That is an SRM. The test works for any allocation and any number of groups: a 90/10 test that logs 90,500 against 9,500 gives χ² = 27.8 and a p-value around 10⁻⁷, even though the small arm is "only" 500 users short. A three-way test that logs 33,900, 33,300 and 32,800 users gives χ² = 18.2 on two degrees of freedom and p = 0.0001.

A widely used alert threshold is p < 0.001 — deliberately strict, because SRM checks run constantly and real SRM bugs usually produce p-values many orders of magnitude smaller. Our checker also flags anything below 0.01 as worth a second look. Fabijan and colleagues report that about 6% of experiments at Microsoft showed an SRM, so a program that never sees one probably is not checking.

The strict threshold is not timidity. A 20,000-user test split 10,150 / 9,850 gives p = 0.034. Under a 0.05 rule that is an incident; under the SRM convention it passes, and rightly — about one healthy test in thirty looks like that by chance. The strictness is also what makes frequent checking affordable: thirty daily checks at 0.001 carry at most a 3% chance of a false alarm between them.

Why you cannot eyeball it

Whether a split is suspicious depends entirely on how many users are behind it. 52/48 is unremarkable at 1,000 users (p = 0.21); 50.3/49.7 is a blatant SRM at 10 million (χ² = 360). For a 50/50 test, these are the splits at which the p < 0.001 alert fires:

Total usersSplit that triggers the alert
10,00051.6% / 48.4%
100,00050.5% / 49.5%
1,000,00050.16% / 49.84%
10,000,00050.05% / 49.95%

Always run the test on raw counts. Rounded percentages from a dashboard throw away exactly the information the test needs.

The usual suspects

  • Variant-specific redirects that drop slow clients.
  • Bot filtering that classifies the variants differently (changing page structure changes bot heuristics).
  • The treatment crashing or timing out for a subset of devices so their telemetry never lands.
  • Targeting or triggering rules evaluated after assignment instead of before.
  • Data pipelines whose deduplication interacts with variant behavior.
  • The classic: ramping one variant's traffic mid-test without resetting assignment.

How to find the cause

  1. Confirm the configuration. Check the split that was actually deployed, including any ramp changes during the test, and enter that as the expected ratio.
  2. Check the unit. Run the check on the unit you randomize — users if assignment is per user, sessions if it is per session.
  3. Segment the check by day, browser, device, and new versus returning users to find where the imbalance concentrates. A day-one-only SRM points at launch bugs; a Safari-only SRM points at redirects or ITP. With the 0.001 threshold you can check twenty segments and still have at most a 2% chance of a false alarm among them.
  4. Walk the funnel. Compare counts at assignment, at first exposure, and in the analysis table. The step where the ratio first breaks is where the bug lives.
  5. Audit the filters. Bot rules, deduplication, and triggering conditions should all be applied identically to every arm.

What a clean check does not prove

Passing the SRM check means the group sizes are consistent with random assignment. It does not certify the experiment. A bug that drops the same share of users from every arm leaves the ratio intact, and so does one that swaps users between arms. Low-traffic tests can also hide a real mismatch simply because the check has little power: at 10,000 users a split has to reach 51.6 / 48.4 before the alert fires. Treat SRM as a necessary gate, alongside checks that event counts, device mix, and pre-experiment metrics look alike across arms.

What to do with an SRM-affected test

Throw the results away. Not "caveat them", not reweight them — the bias direction and size are unknowable from the inside, which is what makes SRM poisonous. Locate the mechanism, fix it, and rerun. Make the check a standing habit: the significance calculator runs it automatically on the visitor counts you enter and warns you before you read anything else, and before planning a follow-up with the sample size calculator, the split gets checked first. Thirty seconds of chi-square beats four weeks of corrupted inference.

SRM checklist

  • Check the split before reading any metric, on wins as much as on losses.
  • Check daily in the first days of a test, when integration bugs surface.
  • Use raw counts and the split that was actually configured.
  • Alert at p < 0.001; take a second look below 0.01.
  • Never "fix" an SRM by trimming or reweighting the data.

Once the split is clean, read the result with statistical significance, explained in hand — and if you are monitoring a live test, the peeking problem applies to SRM-clean tests just as much.

  1. Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19), 2156–2164.How often sample ratio mismatch occurs, its causes and how to diagnose it.
  2. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.Sample ratio mismatch as a standard trust check on every experiment.