Analyzing an A/B test means answering five questions in order: is the data trustworthy, did the main metric change beyond chance, by how much, did anything else get worse, and what might explain it? Most mistakes come from jumping straight to the second question.
Layer 1: Validity
Before looking at results, check that the experiment ran as designed.
- Sample ratio. Does the split match the plan? A mismatch means users were lost or added unevenly, and results can't be trusted. Use the SRM checker.
- Exposure. Did visitors in B actually see the change? Check for errors, flicker or variants that failed to load on some devices.
- Duration. Did the test reach its planned sample and run full weeks?
- Contamination. Did a sale, outage, tracking change or other test overlap?
If validity fails, stop. Fix and rerun.
Layer 2: The primary metric
This is the one metric you chose in advance to judge the test, usually conversion rate or revenue per visitor. Apply the test you planned: a two-proportion z-test for conversion, Welch's t-test or bootstrap for revenue. The result is a p-value (frequentist) or a probability of B beating A (Bayesian). Our significance calculator gives both for conversion data.
Layer 3: Effect size and uncertainty
Significance tells you whether there's likely a difference. The confidence interval tells you how big it might be. A lift of +12% with an interval of +1% to +23% supports a different decision from +12% with an interval of +10% to +14%.
Ask: is the low end of the interval still worth the cost of the change?
Layer 4: Guardrails
Guardrail metrics catch harm the primary metric might miss:
- average order value (did conversions rise because cheaper items sold?),
- refund or return rates,
- page load time and errors,
- downstream steps (more add-to-carts but the same purchases).
A significant win on the primary metric with a clear loss on a guardrail needs a judgment call, not an automatic ship.
Layer 5: Segments, for ideas only
Splitting results by device, traffic source or new vs returning visitors can generate useful hypotheses. It's weak evidence on its own. Slice the data ten ways and you'll usually find one slice where B "wins" by chance. If a segment result looks important, test it directly in a follow-up.
The output: a decision and a record
Analysis ends with a decision (ship, don't ship, iterate, or retest) and a short record: hypothesis, dates, sample, primary result with interval, guardrails, decision and what you learned. That record is what makes the next test smarter.
For a step-by-step version with numbers, see how to analyze A/B test results.