Is A/B Testing Worth It?

A/B testing is worth it when the expected gain from finding better versions exceeds the cost of tools, time and the traffic spent on losers. For stores with steady traffic and real decisions to make, it usually is. For small sites testing small tweaks, it usually isn't, and there are better uses of the time.

The costs

• Tooling. From free open-source options to enterprise platforms priced by quote.
• Build time. Designing, building and checking each variant.
• Analysis time. Small for a single test, more if results are muddy.
• Traffic cost. While a test runs, part of your traffic sees the worse version. If B is worse, that's lost sales; if B is better, the control half is the lost opportunity.
• Waiting. A test that needs six weeks delays the decision by six weeks.

The benefits

• Wins that last. A confirmed 5% lift on checkout applies to all future traffic.
• Losses avoided. A redesign that would have cut conversion gets caught.
• Learning. Each result tells you something about your customers that improves future ideas.

The traffic threshold

The decisive factor is how small an effect you can detect. At a 2.5% conversion rate, 95% confidence, 80% power and a month of testing on a 50/50 split:

• 1,000 visitors a day: detectable lift about 21%.
• 300 visitors a day: detectable lift about 42%.

Most individual changes don't move conversion by 20%, let alone 40%. So at low traffic, a testing program built on small tweaks will produce a stream of inconclusive results. That's the most common reason people decide testing "doesn't work".

Run your own numbers in the MDE calculator.

A quick expected-value check

Say a store makes $60,000 a month and a test idea has a one-in-three chance of a 10% lift. If it works and you detect it, the gain is roughly $6,000 a month going forward. If the test costs a week of someone's time plus tool fees, the expected value is clearly positive, if your traffic can detect a 10% lift. If it can only detect 30%, the probability of a clear result collapses and so does the value.

Making it worth it at lower traffic

• Test bolder changes. New offers, new page structures, new pricing.
• Test where traffic concentrates. Product templates across all products, not one product.
• Use frequent metrics. Add-to-cart rates are several times higher than purchase rates, so they need far fewer visitors.
• Accept 90% confidence for low-risk decisions, and say so.
• Fix the basics first without testing. The CRO checklist lists the usual suspects.

The honest answer

Testing is worth it when you use it for decisions that matter, size it properly and accept its answers. It isn't worth it as a ritual applied to every small change. The math to tell the two apart is free and takes a minute.