What the checker does
It compares the visitors you observed in each group with what the planned split predicts, using a chi-square goodness-of-fit test:
The default example has 50,410 and 49,210 visitors on a planned 50/50 split. Expected is 49,810 each. χ² is 14.45 and p is 0.00014, well under the 0.001 threshold. Something is wrong with that test.
Compare 5,050 versus 4,950: p is 0.32, a perfectly normal wobble.
You can add up to eight groups and use any split, such as 90/10 for a cautious rollout or 34/33/33 for a three-way test.
Why SRM matters so much
The point of randomization is that both groups are alike in every way except the change. If one group is missing visitors, it's missing a particular kind of visitor: people with slow connections, people who bounced during a redirect, people on a browser where a script failed. Those people convert differently, so the comparison is biased, and in an unknown direction.
Where to look when you find one
- Redirect tests. Split URL tests lose users who leave during the redirect. Compare when each group's tracking fires.
- Page speed. If the variant loads a heavy script, tracking may fire later and miss quick bounces.
- Bots and internal traffic. Filters applied after assignment can remove more users from one side.
- Mid-test changes. Changing the traffic split or editing the variant partway through often breaks the ratio.
- Assignment bugs. Check that users keep the same variant across pages and visits.
Run this check while the test is live, not only at the end. Catching SRM on day three saves weeks.
Common questions
What is sample ratio mismatch?
SRM is when the number of users in each group differs from the planned split by more than chance can explain. A test set to 50/50 that ends 50,410 versus 49,210 has a mismatch (p about 0.0001). It usually means users are being lost, double-counted or assigned unevenly in one group.
What p-value threshold should I use for SRM?
Many experimentation teams use p < 0.001 rather than 0.05, because you run this check on every test and want few false alarms. A strict threshold still catches real problems, since broken setups tend to produce very small p-values once traffic builds up.
What causes SRM?
Common causes are a redirect in one variant that loses some visitors, a variant that loads slower so the tracking fires less often, bot traffic hitting one version, filtering applied after assignment, caching that serves the old page, and changes to the split during the test.
Can I just fix the numbers and use the results?
No. SRM means the groups are no longer comparable, and you can't tell which users went missing or why. Find the cause, fix it, and rerun the test.
Does a 52/48 split always mean SRM?
It depends on volume. With 1,000 visitors a 52/48 split is normal noise. With 100,000 visitors it is a near-certain mismatch. That's why you need a test, not a rule of thumb.