What CRO actually is
Conversion rate optimization means increasing the share of visitors who do what you want: buy, sign up, start a trial. In practice it's a loop:
- Find where and why people drop off.
- Form a specific idea about what would help.
- Make the change.
- Measure whether it worked.
- Keep what works and learn from what didn't.
A/B testing lives in step 4. It's the most reliable way to measure the effect of a change on a live site, because it controls for everything else that changes at the same time: seasons, campaigns, news, your competitors' sales.
Step 1: Research before ideas
The best test ideas come from evidence, not brainstorms. Useful sources, roughly in order of effort:
- Funnel data. Where is the biggest drop: product page to cart, cart to checkout, or within checkout?
- Session recordings and heatmaps. Watch twenty recordings of people who left the cart. Patterns show up fast.
- On-site surveys. Ask exit visitors one question: "What stopped you from buying today?"
- Customer service logs. The questions people email about are the answers your pages don't give.
- Usability tests. Five people trying to buy something will find your worst problems.
Step 2: Write a hypothesis
A useful hypothesis names the change, the expected effect and the reason:
Because exit surveys say shipping costs surprise people at checkout, showing the shipping cost on the product page will increase completed orders.
The reason matters. If the test loses, the reason tells you what you've learned; if it wins, it points to related ideas.
Step 3: Prioritize
You'll have more ideas than traffic. Score them on expected impact, confidence in the evidence and ease, then work down the list. Our testing roadmap guide includes an ICE scorer.
Before committing, check each idea against your minimum detectable effect. Ideas whose realistic effect is far below it won't produce a clear result.
Step 4: Design the test
- One primary metric. Usually orders or revenue per visitor. Add a few guardrail metrics (refunds, average order value) to catch harm.
- Sample size and duration, decided upfront. Use the sample size and duration calculators.
- The right audience. Only include visitors who can see the change.
- QA every variant on every major browser and device before launch.
Step 5: Run it without interfering
Check during the test for breakage: errors, broken layouts, a lopsided split (use the SRM checker). Don't check for a winner. Stopping early on a good day is the most common way teams fool themselves, as the A/A simulator shows.
Avoid launching other big changes on the same pages mid-test.
Step 6: Analyze honestly
At the planned end, run the numbers through the significance calculator and look at:
- the confidence interval for the lift, not just the p-value,
- guardrail metrics,
- whether the result makes sense given the hypothesis.
Segment results (mobile vs desktop, new vs returning) are useful for new hypotheses but weak as proof. Slice a test ten ways and one slice will "win" by chance.
Step 7: Roll out and record
Ship winners. Keep a simple log of every test: hypothesis, dates, sample, result, interval, decision. After a year, that log is the most valuable thing your CRO program owns, because it tells you what your customers respond to.
Common reasons tests fail
- Too little traffic for the effect size.
- Peeking and stopping early.
- Changes too small to matter.
- Testing on the wrong page (a checkout fix measured against all sessions).
- Novelty effects: returning visitors react to anything new for a week or two.
More in common A/B testing mistakes.
Common questions
What is the difference between CRO and A/B testing?
CRO is the overall practice of improving the share of visitors who take a desired action. It includes research, usability fixes, copy, design and pricing. A/B testing is one method inside CRO, used to measure whether a specific change improves the outcome.
Do I need A/B testing to do CRO?
No. Fixing a broken checkout or a confusing shipping policy doesn't need a test. Testing earns its keep when a change is uncertain, costly to reverse, or likely to have side effects you can't predict.
How much traffic do I need for A/B testing?
It depends on your conversion rate and the size of effect you need to see. As a rough guide, a store converting at 2% to 3% needs tens of thousands of visitors per variant to detect a 10% relative lift. Lower traffic sites should test bigger changes or test higher-frequency actions like add-to-cart.
What should I test first?
Start where traffic is high and the problem is clear from research, typically the product page, cart or checkout. Avoid testing button colors first; such changes rarely move conversion enough to detect.