The right test depends on what you're measuring. Match the test to the shape of the metric, then match the procedure to how you'll look at the data.
Decision table
| Metric | Example | Test |
|---|---|---|
| Yes/no per user, large counts | Converted or not | Two-proportion z-test or 2x2 chi-square |
| Yes/no per user, small counts | 3 of 40 vs 11 of 42 | Fisher's exact test |
| Yes/no, three or more variants | A vs B vs C | Chi-square across all, then pairwise tests with a correction |
| Continuous per user | Revenue per visitor, order value | Welch's t-test; bootstrap if very skewed or small |
| Count per user | Pages viewed, items in cart | Welch's t-test at large n; Poisson or negative binomial models otherwise |
| Ratio of two totals | Revenue per order, clicks per session | Delta method or bootstrap at the user level |
| Probability B is best | Any binary metric | Beta-Binomial Bayesian model |
Conversion rates: z-test or chi-square
Most store tests measure whether a visitor converted. The two-proportion z-test is the standard tool, and the 2x2 chi-square test gives the identical p-value. Our significance calculator shows both. The details are in the significance test for A/B testing.
Small counts: Fisher's exact
The z-test relies on a normal approximation that gets shaky when a group expects fewer than about five conversions. Fisher's exact test computes the probability directly. Example: 3 of 40 versus 11 of 42 gives p = 0.025 from the z-test but 0.038 from Fisher's test. Both are significant here, but in borderline cases the choice matters.
Revenue: Welch's t-test, carefully
Revenue per visitor is mostly zeros (non-buyers) with a long tail of large orders. With tens of thousands of visitors per group, Welch's t-test (which doesn't assume equal variances) works acceptably. Watch out for:
- Outliers: one wholesale order can swing the mean. Many teams cap values at a high percentile, decided before the test.
- Smaller samples: use a bootstrap confidence interval.
- Medians: tests on medians (such as Mann-Whitney) answer a different question. They don't tell you whether total revenue went up.
Ratio metrics: the unit of analysis matters
Metrics like revenue per order or click-through per session divide two totals, and the denominator varies by user. Treating each order or session as independent understates the uncertainty, because one user contributes many. Analyze at the user level with the delta method or a bootstrap.
Multiple variants and metrics
Comparing three variants with control at α = 0.05 each gives a much higher chance of at least one false winner. Use a correction (Bonferroni is simple and conservative) or plan a single primary comparison. Similarly, decide on one primary metric; with ten independent metrics at 5%, the chance of at least one false positive is about 40%.
How you look matters as much as which test
All the fixed-sample tests above assume you analyze once, at a planned sample size. If you check results daily and stop on a good day, the real false positive rate climbs; see the A/A simulator. If you need to monitor continuously, use a sequential test designed for that.
Common questions
Should I use a t-test or a z-test for conversion rates?
For conversion rates with typical online sample sizes, the two-proportion z-test (or chi-square) is standard. A t-test on 0/1 data gives nearly the same answer at large samples, so the choice rarely changes the conclusion.
What test should I use for average order value?
Welch's t-test is a reasonable default for large samples. Order values are skewed, so with smaller samples or extreme outliers, consider a bootstrap confidence interval or capping extreme values before testing.
Should I use Bayesian or frequentist methods?
Either works if applied correctly. Frequentist tests give p-values and confidence intervals with well-understood error rates for a fixed sample. Bayesian methods give direct probabilities like "chance B beats A" but depend on the prior. Many teams look at both.