Is A/B Testing Really Reliable?

A properly run A/B test is about the most reliable evidence a business can get about cause and effect. The catch is "properly run". Many real-world tests are too small, stopped early, broken in ways no one checked, or analyzed by hunting through metrics until something looks good. Those results are unreliable, and a lot of disappointment with testing comes from them.

What makes a test trustworthy

Randomization is the source of A/B testing's strength. When visitors are assigned by chance, the groups differ only in the change and in luck. Statistics handles the luck. So a well-run test answers "did this change cause a difference?" more convincingly than any before-and-after comparison, survey or expert opinion.

Where reliability breaks down

Too little traffic. An underpowered test has two problems: it usually misses real effects, and when it does find a "winner", the measured lift tends to be exaggerated, because only lucky overestimates cross the significance line. Use the sample size calculator before launch.

Peeking. Checking results repeatedly and stopping at the first significant moment inflates false positives. With 10 checks at α = 0.05, the real false positive rate is about 19%. The A/A simulator demonstrates this.

Broken splits. If one variant loses visitors to a slow redirect or a script error, the groups are no longer comparable. Sample ratio mismatch is the warning sign; check with the SRM checker.

Too many metrics and segments. Track 20 metrics at α = 0.05 and you have about a 64% chance that at least one shows a false positive. Pick one primary metric in advance.

Novelty and change aversion. Returning visitors may react to anything new, positively or negatively, for a week or two. Run tests long enough for this to settle, and check whether the effect is shrinking over time.

External validity. A test in December tells you about December shoppers. Results may not transfer to other seasons, traffic sources or markets.

A realistic picture

Even well-run tests at α = 0.05 and 80% power will:

• declare a winner in about 5% of cases where there's no real effect,
• miss about 20% of real effects of the planned size, and more of smaller effects.

Across a program where most ideas don't work, that means a noticeable share of "winners" are false. That's not a reason to stop testing; it's a reason to treat single results as strong evidence rather than certainty, and to retest surprising or high-stakes wins.

Making your program more reliable

1. Plan sample size and duration, then stick to them.
2. Check the split on every test.
3. Use one primary metric and a few guardrails.
4. Report confidence intervals, not just "significant".
5. Retest results that are surprising or will drive big decisions.
6. Keep a log, so patterns across tests carry more weight than any single one.

Done this way, A/B testing is very reliable. Done casually, it's a random number generator with a dashboard.