Sample Ratio Mismatch (SRM): The Silent A/B Test Killer

Sample ratio mismatch (SRM) is the silent killer of A/B tests: a traffic split that deviates from its configuration by more than chance allows, revealing that users were sorted into groups by some systematic force rather than the coin flip your statistics assume. This guide covers detection, causes, and the uncomfortable question of what to do with an affected test.

Why a small imbalance is a big deal

The threat is not the imbalance itself — tests tolerate unequal group sizes fine. The threat is what the imbalance proves: some mechanism is removing or adding users non-randomly. Missing users are never a random sample (they are the slow connections, the bot-filtered, the crash victims), so the groups no longer differ only by treatment. A 0.5% assignment bias can produce metric deltas larger than a typical true treatment effect — in either direction.

Detection

A chi-square goodness-of-fit test compares observed counts to the configured split; the SRM checker does it in seconds with a plain-English verdict. The industry alert threshold is p < 0.001 — deliberately strict, because SRM checks run constantly and real SRM bugs usually produce p-values many orders of magnitude smaller. At scale, roughly 6% of experiments at large tech companies trigger SRM alerts, so a program that never sees one probably is not checking.

The usual suspects

Variant-specific redirects that drop slow clients. Bot filtering that classifies the variants differently (changing page structure changes bot heuristics). The treatment crashing or timing out for a subset of devices so their telemetry never lands. Targeting or triggering rules evaluated after assignment instead of before. Data pipelines whose deduplication interacts with variant behavior. And the classic: ramping one variant's traffic mid-test without resetting assignment.

What to do with an SRM-affected test

Throw the results away. Not "caveat them", not reweight them — the bias direction and size are unknowable from the inside, which is what makes SRM poisonous. Locate the mechanism (segment the SRM check by browser, device, and day to find where the imbalance concentrates — a day-one-only SRM points at launch bugs; a Safari-only SRM points at redirects or ITP), fix it, and rerun. Make the check a standing habit: before reading any result in the significance calculator or planning a follow-up with the sample size calculator, the split gets checked first. Thirty seconds of chi-square beats four weeks of corrupted inference.