Most bad test results don't come from bad ideas. They come from a handful of process mistakes that turn noise into false winners or hide real effects. Here are twelve, roughly in order of how much damage they do.
1. Stopping when it looks good
Checking daily and stopping on the first significant result is the classic error. With 10 equally spaced checks at α = 0.05, about 19% of tests with no real difference produce a "winner". Fix: set sample size and end date before launch. See it in the A/A simulator.
2. Running underpowered tests
A test sized for a 30% lift will miss most real effects of 5% to 10%, and the winners it finds will have inflated lifts. Fix: size with the sample size calculator; if the number is out of reach, test something bolder.
3. Ignoring sample ratio mismatch
A 50/50 test that delivers 50.6/49.4 on 100,000 users is broken, even though it looks close. Fix: run every test through the SRM checker, during and after.
4. Fishing through metrics
Track 20 metrics and the chance that at least one shows a false positive at α = 0.05 is about 64%. Fix: pick one primary metric before launch. Treat the rest as guardrails or idea generators.
5. Declaring segment winners
"B lost overall but won on iOS" is usually noise. Fix: use segments to generate hypotheses; confirm with a dedicated test.
6. Reading p-values as probabilities that B is better
p = 0.03 doesn't mean a 97% chance B wins. Fix: report the confidence interval and, if you want a direct probability, a Bayesian probability to beat. The significance calculator shows both.
7. Ignoring the size of the effect
A significant lift of +0.4% with an interval from +0.1% to +0.7% may not pay for the change. Fix: decide what lift is worth shipping, and compare it with the low end of the interval.
8. Testing tiny changes on low traffic
Button colors and single words rarely move conversion enough to detect on a modest store. Fix: check ideas against your MDE; bundle small changes into bolder variants.
9. Counting the wrong visitors
Measuring a checkout change against all site sessions dilutes the effect with people who never reached checkout. Fix: include only users who could see the change, and use the conversion rate at that step as your baseline.
10. Running partial weeks or atypical periods
A test from Wednesday to Monday, or across a flash sale, measures something unusual. Fix: whole weeks, no major promotions unless that's what you're testing.
11. Letting the variant be slower
A client-side test that adds half a second of load time or makes the page flicker is testing speed as much as design. Fix: QA load times and rendering for every variant on real phones.
12. Not writing anything down
Six months later, nobody remembers what was tested or why. Fix: keep a log: hypothesis, dates, sample, result with interval, decision, lesson. It's the cheapest improvement you can make.
A quick pre-launch checklist
- Hypothesis written, with a reason.
- One primary metric and a few guardrails.
- Sample size and end date decided.
- Variants checked on mobile and desktop, fast and flicker-free.
- Only eligible visitors included.
- A plan to check the split during the test.
Common questions
What is the most common A/B testing mistake?
Stopping a test as soon as it looks significant. It feels efficient but sharply increases the chance of a false winner. Decide the sample size in advance and analyze at the end.
How do I know if my test was too small?
Check the confidence interval. If it spans from a meaningful loss to a meaningful gain, the test couldn't distinguish between them. Before the next test, use a sample size or MDE calculator.