How to Analyze A/B Test Results, Step by Step

Here's the checklist we'd follow at the end of any conversion test, worked through with an example. It takes about ten minutes.

Example: a store tested a shorter product description (B) against the current one (A).

• A: 1,200 visitors, 48 orders
• B: 1,180 visitors, 61 orders

Step 1: Confirm the test ran as planned

• Did it reach the planned sample size? (Here, suppose the plan was 1,200 per group. Yes.)
• Did it run whole weeks? (Two full weeks. Yes.)
• Anything unusual during the test? (No sales, outages or tracking changes.)

Step 2: Check the split

The plan was 50/50, and we got 1,200 vs 1,180. In the SRM checker that gives p = 0.68: no mismatch. If p were below 0.001, we'd stop here and investigate.

Step 3: Compute the rates and lift

• A: 48 / 1,200 = 4.00%
• B: 61 / 1,180 = 5.17%
• Relative lift: +29%

That looks big. Now the important part.

Step 4: Test for significance

Enter the counts in the significance calculator:

• two-sided p-value: 0.17
• 95% confidence interval for the lift: roughly -11% to +87%

Not significant at 5%. The interval is very wide: the data is consistent with B being somewhat worse or dramatically better.

Step 5: Read the Bayesian probability

The same calculator reports a 91% probability that B's true rate is higher than A's. That's encouraging, but notice how it differs from the p-value. They're answering different questions, and neither says "B is 29% better".

Step 6: Check guardrails

Did average order value change? Did returns increase? For a description change, also check whether customer questions about the product rose. Suppose everything is flat.

Step 7: Decide

Options here:

1. Keep testing to the size needed to detect a smaller effect. A good plan if the decision matters.
2. Ship B on the grounds that it's cheap, likely better (91%) and unlikely to be much worse. Reasonable for a low-risk change, as long as you report it honestly as "not significant, probably better".
3. Don't ship and move on. Reasonable if the change has other costs.

What you shouldn't do is report "B lifts conversion by 29%". The data doesn't support that number.

Step 8: Write it down

Record the hypothesis, dates, counts, p-value, interval, probability to beat, guardrails, decision and a one-line lesson. In this case: "Shorter descriptions may help; underpowered. Retest on the full catalog template."

The same steps for a significant result

When a test is significant, steps 6 and 7 still apply. Check that the low end of the confidence interval is worth the cost of the change, and that nothing important got worse. Significance is the start of the decision, not the end.