What the simulator does
For each simulated test it draws visitors for two groups from the same true conversion rate. It checks the result as many times as you set, using a two-sided two-proportion z-test each time, and records:
- whether the test was significant at the final look, and
- whether it was significant at any look, which is what happens if you stop the first time p falls below alpha.
It also plots the final p-values. When there's no real difference, p-values are spread evenly between 0 and 1, so about 5% fall below 0.05.
Try this
- Leave "Times you check results" at 1 and run it. The false winner rate lands near 5%.
- Set it to 14, as if you checked every day for two weeks. Run again. The rate of tests that showed a "winner" at some point jumps to somewhere around 20% to 25%.
- Notice that the final-look rate stays near 5%. Looking doesn't change the data. Acting on what you see does.
The simulation uses a seeded random generator, so the same inputs and seed always reproduce the same numbers. Our test suite checks that one look gives about 5% and that ten looks give about 19%, in line with the published figure.
What this means for your tests
If you've been checking a dashboard daily and calling tests as soon as they turn green, a good share of your past winners were probably noise. That's not a reason for despair. It's a reason to:
- decide sample size and duration before launch (sample size, duration),
- look during the test only for breakage, not for a winner,
- or switch to a sequential method if you need early stopping.
Common questions
What is an A/A test?
An A/A test splits traffic between two identical versions. Since nothing differs, any significant result is a false positive. Teams run real A/A tests to validate a new testing tool, check for sample ratio mismatch and confirm that tracking works.
How often should an A/A test come up significant?
About as often as your significance level. At alpha = 0.05, roughly 1 in 20 A/A tests will show a significant difference if you look only once at the end. That's expected, not a sign of a broken tool.
Why does peeking increase false positives?
Each time you check, you get another chance for random noise to cross the significance line. If you stop at the first crossing, those chances add up. With 10 equally spaced checks at alpha = 0.05, about 19% of tests with no real difference will show a winner at some point, a figure first published by Armitage, McPherson and Rowe in 1969.
How can I check results during a test without this problem?
Use a method built for it, such as a group sequential design with alpha spending, or the always-valid sequential tests some platforms use. Or check only for problems like SRM or broken pages, and leave the decision for the planned end.