How to read the output
The calculator answers two related questions.
The frequentist question: if A and B actually convert at the same rate, how surprising is the gap you observed? That's the p-value. With the default example (320 of 10,000 versus 380 of 10,000) the p-value is 0.021, so a gap this big would turn up about 2% of the time by luck alone. At a 5% threshold, that counts as significant.
The Bayesian question: given the data, how likely is it that B's true rate is higher? For the same numbers the answer is 99.0%.
These numbers are not the same thing and won't match. A p-value of 0.021 doesn't mean a 97.9% chance that B is better.
The confidence interval matters more than the p-value
In the example the observed lift is +18.8%, but the 95% confidence interval runs from about +2.6% to +37.4%. Both ends are consistent with the data. If a 3% lift wouldn't pay for the change, you don't have the answer you need yet, even though the test is "significant".
The interval for the absolute difference uses the unpooled standard error. The interval for the relative lift uses the log of the ratio of the two rates, which behaves better than a straight percentage when rates are small.
Methods used
The probability that B beats A uses a closed-form sum, so it isn't a simulation estimate. The credible interval and expected loss come from 40,000 random draws with a fixed seed, so the same inputs always give the same output. All of this is checked against independent reference values; see the methodology.
Before you trust any result
- Check the split. Run the visitor counts through the SRM checker. A broken split invalidates everything else.
- Stick to the plan. If you didn't reach the sample size you planned, or you looked every day and stopped on a good day, the p-value overstates your evidence.
- Check one metric. If you tested ten metrics and one came up significant, expect that to happen by chance about 40% of the time at a 5% threshold.
- Look at segments only to generate ideas. A variant that "wins on mobile" after losing overall is usually noise.
Common questions
What does statistically significant mean in an A/B test?
It means that if the two versions truly performed the same, a difference as large as the one you saw would be rare, rarer than your chosen threshold (usually 5%). It does not mean there is a 95% chance B is better, and it says nothing about whether the lift is big enough to matter.
Is the z-test the same as a chi-square test?
For two groups and a yes/no outcome, yes. The pooled two-proportion z-test and the 2x2 chi-square test without continuity correction give the same p-value, because the chi-square statistic equals z squared. The calculator shows both numbers.
What is the probability that B beats A?
It is a Bayesian quantity. Starting from a flat Beta(1,1) prior on each conversion rate, the calculator updates with your data and computes the exact probability that B's true rate is higher than A's. It is often what people mean when they ask whether B is better, but it depends on the prior and doesn't replace a stopping rule.
My result is significant but the confidence interval is huge. What now?
Ship it if any value in the interval would still be acceptable to you. If the low end is a lift too small to justify the change, or the decision is expensive, run a confirmation test or keep the test running to the sample size you planned.
Why is my p-value different from my testing tool's?
Tools differ. Some use sequential methods that adjust for continuous monitoring, some use t-tests on revenue, some count sessions instead of users, and Bayesian tools may use different priors. Check that you are feeding the same counts, then read the tool's documentation for its method.