An A/B test shows two different versions to two random groups and asks which performs better. An A/A test shows the same version to both groups. Since nothing differs, it's not testing an idea; it's testing your testing setup.
Why run a test where nothing changes?
Because it checks the machinery:
- Is the split working? If you planned 50/50 and got 52/48 on 100,000 users, something in assignment or tracking is off. The SRM checker will tell you.
- Is tracking the same in both groups? A tag that fires differently in one group shows up as a difference in conversion.
- Are the statistics behaving? Across many A/A tests, false positives should appear at about your significance level.
What to expect from an A/A test
At α = 0.05, about 1 in 20 A/A tests will show a "significant" difference by pure chance. That's not a malfunction; it's the definition of a 5% false positive rate. If you run one A/A test and it's significant, look into it, but don't panic.
What should worry you:
- the visitor split is off (sample ratio mismatch),
- the "difference" is very large,
- repeated A/A tests come up significant far more often than 5%.
See it for yourself
The A/A simulator runs thousands of simulated A/A tests in your browser. Looking once at the end, about 5% come up significant. Set it to check results 14 times and stop at the first significant look, and the share of false winners rises to over 20%. That's the peeking problem, and an A/A simulation is the clearest way to see it.
A/A/B tests
Some teams add a second control to a regular test: A, A' and B. If A and A' differ a lot, the test has a problem. It costs traffic, since three groups need more visitors than two, so it's mostly used when validating a new setup.
When to run a real A/A test
- When you adopt a new testing tool.
- After major changes to tracking, analytics or your site's tech stack.
- When results from your tests seem unstable or too good to be true.
Size it like a normal test, run it for a week or two, and check both the split and the conversion difference. If it passes, you can trust the setup with real decisions.
Common questions
What is an A/A test used for?
To check that a testing setup works: traffic splits as planned, both groups are tracked the same way, and the statistics produce false positives at the expected rate. It's most useful when adopting a new tool or changing tracking.
My A/A test showed a significant difference. Is my tool broken?
Not necessarily. At a 5% significance level, about one A/A test in twenty will be significant by chance. It's worth investigating if it happens much more often, if the split itself is off, or if the difference is huge.