"A/B test" covers a family of related designs. They all randomly split traffic and compare outcomes, but they differ in how many versions there are, how traffic is divided and what's being changed.
Classic A/B test
Two versions, usually 50/50: the current page (A) and one change (B). It's the simplest design and needs the least traffic per answer, which makes it the default for most stores.
A/B/n test
More than two versions at once: A against B, C and D. Useful when you have several distinct ideas for the same element. The cost: each extra version needs its own full-size group, so a four-way test needs twice the traffic of an A/B test. Comparing several variants with control also raises the chance of a false winner unless you adjust the significance level. The sample size calculator has a Bonferroni option for this.
Split URL test
Version B lives at a different URL, and visitors are redirected to it. Handy for testing a full page redesign or a different checkout flow that's hard to build as an in-page change. Watch for visitors lost during the redirect, which shows up as sample ratio mismatch. For SEO, use temporary redirects and a canonical tag pointing to the original. More in A/B vs split testing.
Multivariate test (MVT)
Several elements change at once, in every combination. Two headlines times two images times two button labels gives eight versions. MVT can reveal interactions (this headline works only with that image), but it splits traffic eight ways. Unless you have very high traffic, run sequential A/B tests instead.
A/A test
Both groups see the same version. Any significant difference is a false positive. A/A tests check that your tool splits traffic evenly and tracks correctly before you trust it with real decisions. See A/A vs A/B testing and try the A/A simulator.
Multi-armed bandit
Instead of a fixed split, traffic shifts toward whichever version is performing better as data comes in. Bandits earn more during the test but give a less precise estimate of the difference. They suit short campaigns (a holiday banner) better than long-term decisions. Our AI in testing guide covers them.
Client-side vs server-side
Client-side tests change the page in the visitor's browser with JavaScript. They're easy for marketers to set up, but can cause flicker and slow pages. Server-side tests decide the version before the page is sent, which is faster and works for deeper changes like pricing logic, search ranking or app features, but usually needs developers.
Price tests
A special case where conversion is expected to move and the decision rests on profit per visitor. They carry legal and trust considerations that other tests don't. See the price testing guide.
Which to pick
| Situation | Design |
|---|---|
| One clear idea, modest traffic | Classic A/B |
| Several competing ideas, high traffic | A/B/n with correction |
| Whole-page or flow redesign | Split URL |
| Interactions between elements matter, very high traffic | Multivariate |
| New tool or tracking setup | A/A first |
| Short campaign, earn while testing | Bandit |
Common questions
What is the difference between A/B and multivariate testing?
An A/B test compares whole versions. A multivariate test changes several elements at once in every combination (for example, two headlines times two images gives four versions) to see how each element and their combinations affect results. Multivariate tests need far more traffic.
Which type of A/B test should a small store use?
A classic two-version A/B test with a bold change. It needs the least traffic per answer. Avoid multivariate tests and many-variant tests until traffic is high.