For an A/B test on a conversion rate (did the visitor buy or not), the standard significance test is the two-proportion z-test. It asks how likely a gap as large as yours would be if both versions truly converted at the same rate. The 2x2 chi-square test is the same test in different clothing and gives the same p-value.
How the z-test works
You have visitors (n) and conversions (x) for each version. The test:
- Calculates each conversion rate: p_A = x_A / n_A and p_B = x_B / n_B.
- Pools both groups to estimate the shared rate you'd expect if there were no difference.
- Works out how much the gap between two such rates would normally wobble by chance, given the group sizes (the standard error).
- Divides the observed gap by that wobble to get z.
- Converts z into a p-value using the normal distribution.
A z of about ±1.96 corresponds to p = 0.05 for a two-sided test.
A worked example
A store tests a new product page:
- A: 25,000 visitors, 500 orders (2.00%)
- B: 25,000 visitors, 560 orders (2.24%)
That's a 12% relative lift. The z-test gives z = 1.86 and p = 0.062. At the usual 5% level, that's not significant: a gap this big would happen about 6% of the time by chance alone.
Interestingly, a Bayesian analysis of the same data puts the probability that B is better at about 96.9%. The two numbers answer different questions, which we explain in how to analyze results. You can check both in the significance calculator.
Reading the p-value correctly
A p-value of 0.062 means: if A and B were truly identical, data at least this extreme would turn up 6.2% of the time.
It does not mean:
- a 6.2% chance that B isn't better,
- a 93.8% chance that B is better,
- the effect is small or large.
For size, look at the confidence interval for the lift. In the example it runs from about -0.6% to +26%, which is an honest picture: probably positive, possibly large, not yet nailed down.
Two-sided or one-sided?
A two-sided test asks "is B different from A?" and catches harm as well as improvement. A one-sided test asks only "is B better?", which halves the p-value for a positive result (0.031 in the example) but can't flag a loss. Most teams use two-sided tests so that a harmful change gets noticed.
When the z-test isn't right
- Small counts: use Fisher's exact test. With 3 of 40 converting against 11 of 42, the z-test gives p = 0.025 and Fisher's exact test gives 0.038.
- Revenue or order value: these aren't yes/no outcomes. Use a t-test or a bootstrap. See t-test vs A/B test.
- Repeated checking: if you plan to look at results as they come in, use a sequential method designed for it.
For a full decision table by metric type, see which test to use for A/B testing.
Common questions
What p-value is significant in an A/B test?
By convention, p below 0.05, meaning a 5% significance level or 95% confidence. Some teams use 0.10 for low-risk changes or 0.01 for high-stakes ones. Choose before the test, not after.
Is the chi-square test the same as the z-test for A/B testing?
For two groups and a yes/no outcome, yes. The 2x2 chi-square test without continuity correction gives exactly the same p-value as the pooled two-proportion z-test, because chi-square equals z squared.
When should I use Fisher's exact test instead?
When counts are small, roughly when any group would expect fewer than five conversions or non-conversions. Fisher's test doesn't rely on the normal approximation.