Use A/B testing when three things are true: you genuinely don't know whether a change will help, the change matters enough to be worth measuring, and you have enough traffic to detect the effect you expect. If any of those is missing, there's usually a better option than a test.
Test when the outcome is uncertain
Good test candidates are changes where reasonable people disagree or where there could be side effects:
- a new product page layout,
- moving shipping costs earlier in the funnel,
- a different offer (free shipping vs a discount),
- a price change,
- removing steps from checkout,
- a new navigation structure.
Test when being wrong is expensive
A redesign that quietly lowers conversion by 8% can cost far more than the redesign did. Testing is insurance: you find out on a fraction of traffic before committing.
Test when you have the traffic
Before planning, run your numbers through the MDE calculator. Some reference points at a 2.5% conversion rate, 95% confidence and 80% power, with traffic split 50/50 for about a month:
| Daily visitors | Smallest detectable lift |
|---|---|
| 300 | about 42% |
| 1,000 | about 21% |
If the realistic effect of your idea is well below the detectable lift, the test will almost certainly end inconclusive.
Don't test these
- Bug fixes. If the add-to-cart button fails on Safari, fix it.
- Legal, accessibility or security requirements. You're doing them regardless.
- Obvious improvements with negligible risk, such as compressing images or fixing a typo.
- Changes too small to detect. Ship or skip them on judgment.
- Decisions already made. If the brand refresh is happening no matter what, a test is theater. (You might still test how to roll it out.)
Be careful with timing
Avoid starting or ending tests during atypical periods (major sales, holidays, stockouts, big PR moments) unless you're specifically testing for those periods. Results from Black Friday rarely carry over to February.
Alternatives when testing isn't practical
- Usability tests with five to eight people find big problems fast.
- Before-and-after with care: compare against the same period last year and against a page you didn't change.
- Surveys to rank which problems matter most.
- Test upstream metrics such as add-to-cart, which happen more often than purchases, then watch purchases after rollout.
Testing is a powerful tool, but it's one tool. The CRO checklist covers many improvements you can make without one.
Common questions
When should you not A/B test?
Don't test bug fixes, legal or accessibility requirements, changes too small to detect with your traffic, or decisions you'll make the same way whatever the result. Also avoid testing during unusual periods like a big sale unless the sale itself is what you're studying.
How much traffic do you need to start A/B testing?
Enough to detect an effect you care about in a few weeks. A store with 1,000 visitors a day and a 2.5% conversion rate can detect a lift of roughly 21% in a month on a 50/50 split. With 300 a day, only lifts around 42% are detectable, so tests need to be bold.