What Is the Principle of A/B Testing?

The principle of A/B testing is the randomized controlled experiment: assign people at random to a control (A) or a change (B), treat the two groups identically in every other way, and attribute any difference in outcomes beyond chance to the change. Everything else in testing practice follows from protecting that principle.

Why randomization is the whole trick

Imagine you show the new page only to visitors from your email list and the old page to everyone else. Email subscribers already like you, so they convert better whatever page they see. The comparison is rigged before it starts.

Random assignment fixes this without you having to know which factors matter. Mobile versus desktop, new versus returning, rich versus broke, in a hurry or browsing: chance spreads all of these evenly between groups on average. You don't need to list them. That's why randomized experiments are trusted where other comparisons aren't.

Rule 1: The only difference is the change

Both groups should get the same prices, the same promotions, the same page speed and the same tracking. If B loads half a second slower because of a heavy script, you're partly testing speed. If an email campaign links only to B, you're testing the campaign.

Rule 2: The split must stay honest

Random assignment only helps if the groups you analyze are the groups you assigned. Redirects that lose visitors, bots that hit one version or filters applied unevenly all undo it. The symptom is a split that doesn't match the plan, called sample ratio mismatch. Check it with the SRM checker on every test.

Rule 3: Decide the question before the answer

Pick the main metric, the sample size and the end date before launch. If you choose the metric after seeing the data, or stop the test on a good day, chance gets extra opportunities to look like an effect. This is the same reason scientists pre-register studies.

Rule 4: Measure chance, don't ignore it

Even with perfect randomization, two groups differ by luck. A significance test or a Bayesian analysis puts a number on how much of the observed difference luck could explain. Our significance calculator shows both.

What the principle doesn't promise

A randomized test tells you the average effect on the people in the test, during the test. It doesn't guarantee:

• the effect will last (novelty wears off),
• it applies to visitors you didn't include,
• you know why it worked.

Those need judgment, follow-up tests and research. But for the narrow question "did this change cause more conversions here and now?", nothing beats a well-run randomized test.

For a gentler introduction, see A/B testing in plain English.