Every A/B test ends in one of five places. Telling them apart is mostly a matter of reading the confidence interval, not just the p-value.
1. Clear winner
B is significantly better, and the confidence interval sits entirely above zero.
Example: 3.00% vs 3.30% on 53,211 visitors each. p = 0.005, the lift's interval is roughly +3% to +18%.
Next step: check guardrail metrics, then ship. Note the low end of the interval: that's the conservative estimate to use in forecasts. Expect the long-run effect to be somewhat smaller than the measured lift, because winners are partly selected for lucky noise.
2. Clear loser
B is significantly worse.
Example: 600 vs 520 orders on 20,000 visitors each (3.00% vs 2.60%), p = 0.015.
Next step: don't ship, and figure out why. A loss is informative: if removing the size guide hurt conversion, sizing uncertainty matters to your customers, and that points to better ideas.
3. Flat result
Not significant, and the confidence interval is narrow around zero.
Example: 6,000 vs 6,020 orders on 200,000 visitors each (3.00% vs 3.01%). p = 0.85, and the lift's interval runs from about -3% to +4%. Whatever the true effect is, it's small.
Next step: choose on other grounds (cost, simplicity, brand) and move on. A flat result needs a lot of traffic; most stores rarely get one.
4. Inconclusive
Not significant, and the confidence interval is wide, including effects you'd care about in both directions. This is the most common outcome for underpowered tests.
Example: 410 vs 405 orders on 20,000 visitors each. p = 0.86, but the lift's interval runs from about -14% to +13%. A real 10% gain or a 10% loss would both fit the data.
Next step: don't claim "no effect". Either extend the test to the planned or a larger sample, accept the uncertainty and decide on judgment, or redesign with a bolder change. Use the MDE calculator to see what your traffic can detect.
5. Invalid
Something broke: a sample ratio mismatch, a variant that failed to load on some browsers, a tracking change mid-test, an overlapping promotion that hit only one group. The numbers can't be trusted, whatever they say.
Next step: find and fix the cause, then rerun. Don't try to "adjust" the data. The SRM checker catches the most common problem.
A sixth outcome to watch for: the surprising winner
Sometimes a change you expected to be neutral wins big. Before celebrating, rule out problems: check the split, look at whether the effect is steady over time or comes from one day, and confirm tracking. If it holds up, consider a confirmation test. Surprising results are more likely than expected ones to be flukes.
Summary table
| Outcome | p-value | Confidence interval | Action |
|---|---|---|---|
| Winner | < α, B better | Above zero | Check guardrails, ship |
| Loser | < α, B worse | Below zero | Don't ship, learn why |
| Flat | ≥ α | Narrow around zero | Decide on other grounds |
| Inconclusive | ≥ α | Wide, crosses zero | More data or bolder test |
| Invalid | Irrelevant | Irrelevant | Fix and rerun |
Common questions
Is a losing A/B test a failure?
No. A clear loser saves you from shipping a harmful change and teaches you something about your customers. The real failures are invalid tests and tests too small to tell you anything.
What's the difference between a flat and an inconclusive result?
Both are non-significant. A flat result has a narrow confidence interval around zero, so any real effect is small. An inconclusive result has a wide interval, so a meaningful effect in either direction is still possible.