What Is A/B Testing, in Plain English?

A/B testing means trying two versions of something at the same time on two groups of similar people, and counting which version gets better results. Version A is usually what you have now. Version B is the change you're curious about. If B does better by more than luck would explain, you keep B.

A bakery example

Say you run a bakery with two doors onto two identical streets. You want to know whether a chalkboard sign saying "Fresh bread at 7am" brings in more customers than the usual "Open" sign.

You could put the new sign up for a week and compare it with last week. But last week had rain, and this week has a street fair. You'd never know whether the sign or the fair did the work.

So instead, you put the old sign at one door and the new sign at the other, and swap them every day. Over a few weeks the weather, the fair and the day of the week affect both doors equally. Whatever difference is left is down to the sign.

That's an A/B test. Online, the "two doors" are created by software that randomly shows each visitor either version A or version B.

Why not just ask people?

Because what people say and what they do often differ. Ask shoppers whether they'd like a cleaner product page and most will say yes. Whether they actually buy more on it is a separate question, and only behavior answers it.

Where luck comes in

Flip a coin 100 times and you won't get exactly 50 heads. You might get 44 or 57. Visitors are similar: even if A and B were identical, one would get a few more sales by chance.

So an A/B test doesn't just ask "which got more?" It asks "is the gap bigger than chance usually produces?" That's what statistical significance measures. Our significance calculator does the arithmetic: you type in visitors and sales for each version, and it tells you how surprising the gap would be if the versions were really the same.

A store example

An online store shows half its visitors the usual checkout and the other half a checkout with the delivery date displayed up top. After three weeks:

• Version A: 10,000 visitors, 320 orders (3.2%)
• Version B: 10,000 visitors, 380 orders (3.8%)

Sixty extra orders sounds convincing, and here it is: a gap that size would show up only about 2% of the time if the two checkouts truly performed the same. The store keeps version B.

If the numbers had been 320 versus 335, the gap would be well within normal luck, and the honest answer would be "we can't tell yet."

The three things that make it work

1. Random split. A computer, not a person, decides who sees which version, so the groups are alike.
2. Same time. Both versions run together, so outside events hit both equally.
3. Enough people. Small groups produce big random swings. The sample size calculator tells you how many you need before you start.

Everything else, from p-values to Bayesian probabilities, is detail on top of those three ideas.