The comparison is a bit like "oven vs dinner". An A/B test is the experiment: randomly split visitors between two versions and measure an outcome. A t-test is a statistical method you can use to analyze the outcome. Sometimes it's the right method; sometimes another test fits better.
What a t-test does
A t-test compares the means of two groups. It looks at the difference between the averages, scales it by how much individual values vary and how many values you have, and asks how surprising that difference would be if the true means were equal. The result is a t statistic and a p-value.
Welch's t-test is the version to use for A/B tests, because it doesn't assume both groups have the same spread.
When a t-test fits A/B data
Use it for metrics that are averages of numbers per user:
- revenue per visitor,
- average order value,
- items per order,
- time on page or session length.
With typical online sample sizes (thousands per group), the t-test is robust even when data isn't normally distributed, thanks to the central limit theorem. Revenue data is very skewed, though, so for modest samples or big outliers a bootstrap or capping extreme values is safer.
When a z-test is the better fit
For conversion rates (yes/no outcomes), the usual tool is the two-proportion z-test or the equivalent chi-square test. It uses the fact that the variance of a proportion is p(1-p), which is more precise than estimating variance from the data as a t-test does.
In practice, with thousands of visitors per group, a t-test on 0/1 conversion data and a z-test give almost identical p-values. The difference matters only for small samples, where Fisher's exact test is better than either.
A quick comparison
| Two-proportion z-test | Welch's t-test | |
|---|---|---|
| Metric | Conversion rate (yes/no) | Mean of a number per user |
| Variance | From p(1-p) | Estimated from the data |
| Small samples | Use Fisher's exact instead | Use bootstrap if data is skewed |
| Typical use | Purchases, sign-ups, clicks | Revenue, order value, time |
A common mistake
Running a t-test on revenue per order while counting each order as independent. A loyal customer with five orders isn't five independent observations. Analyze per user (revenue per visitor) or use a method that accounts for repeat units.
Where to go next
For conversion data, the significance calculator runs the z-test and a Bayesian analysis. For a full decision table by metric type, read which test to use for A/B testing.
Common questions
Can I use a t-test for A/B testing?
Yes, especially for continuous metrics like revenue per visitor or time on page. For conversion rates, a two-proportion z-test is more common, though at large samples a t-test on 0/1 data gives almost the same p-value.
Should I use Student's or Welch's t-test?
Welch's. It doesn't assume both groups have the same variance, which is rarely true when a change affects behavior. When variances happen to be equal, Welch's test loses almost nothing.