Marketers say "A/B test"; statisticians say "two-sample hypothesis test on a randomized experiment". They're describing the same activity with different words. This page translates between the two and walks through one example in both.
Translation table
| A/B testing term | Statistics term | Meaning |
|---|---|---|
| Control (A) | Control group | Current version |
| Variant (B) | Treatment group | Changed version |
| "No difference" | Null hypothesis (H₀) | p_A = p_B |
| "B is different" | Alternative hypothesis (H₁) | p_A ≠ p_B |
| Confidence level, 95% | 1 - α | α = 0.05 is the false positive rate you accept |
| Statistical significance | Rejecting H₀ | p-value below α |
| False winner | Type I error | Rejecting H₀ when it's true |
| Missed winner | Type II error | Not rejecting H₀ when H₁ is true |
| Power, 80% | 1 - β | Chance of catching a real effect of the planned size |
| Minimum detectable effect | Effect size for power calculation | Smallest effect the test is designed to catch |
| Lift | Effect estimate | (p_B - p_A) / p_A |
| Peeking | Optional stopping | Inflates the Type I error rate |
One test, two descriptions
Marketing version. "We tested a new checkout against the old one. We wanted to catch at least a 10% improvement on a 3% conversion rate, so we planned for about 53,000 visitors per version at 95% confidence and 80% power. After four weeks, the new checkout converted at 3.30% vs 3.00%, statistically significant, so we shipped it."
Statistics version. "Users were randomized 1:1 to control or treatment. H₀: p_T = p_C; H₁: p_T ≠ p_C; α = 0.05 two-sided. A sample of n = 53,211 per arm gives 80% power to detect p_T = 0.033 against p_C = 0.030. Observed: 1,596 of 53,211 (3.00%) and 1,756 of 53,211 (3.30%). Two-proportion z-test, p = 0.005. H₀ rejected."
Same experiment, same conclusion. You can reproduce the sample size in the sample size calculator and the p-value in the significance calculator.
Where the vocabularies diverge
"95% confident B is better." Marketing reports often say this when p < 0.05. Statistically that's wrong: a p-value is the probability of data this extreme if H₀ were true, not the probability that B is better. The Bayesian probability that B beats A is the number closer to that phrasing, and it's a different calculation.
"The test was inconclusive." Statistically: we failed to reject H₀. That doesn't mean H₀ is true. The confidence interval tells you which effects remain plausible.
"We reached significance on day 4." Statistically: we performed an unplanned interim analysis. Without a correction, the Type I error rate is no longer α.
Why the translation matters
When you read vendor documentation, academic papers or a statistician's review of your test, they'll use the statistical terms. When you report to the business, you'll use the marketing ones. Knowing both lets you check that nothing got lost in between, especially the meaning of "significant".
Common questions
What are Type I and Type II errors in A/B testing?
A Type I error is declaring a winner when there's no real difference (a false positive); its rate is alpha, usually 5%. A Type II error is missing a real effect (a false negative); its rate is 1 minus power, usually 20%.
What is statistical power in an A/B test?
The probability that the test detects a real effect of a given size. With 80% power for a 10% lift, a true 10% lift will reach significance in about four out of five tests.