Not exactly. Hypothesis testing is a statistical procedure for deciding whether data is inconsistent with a default assumption. An A/B test is a randomized experiment that typically uses a hypothesis test in its analysis step. The test is part of the experiment, not the whole of it.
What hypothesis testing is
A hypothesis test starts with a null hypothesis (no difference), collects data, and calculates how surprising that data would be if the null were true. If it's surprising enough, below a pre-set significance level such as 5%, you reject the null.
It's a general tool used in medicine, manufacturing, psychology and elsewhere. It doesn't care where the data came from.
What an A/B test adds
An A/B test wraps that tool in an experimental design:
- A hypothesis about a change, with a reason: "Showing delivery dates will raise orders because customers worry about timing."
- Randomization, which makes the groups comparable so a difference can be attributed to the change.
- Planning: primary metric, sample size, duration and significance level chosen in advance.
- Execution: running both versions at once, checking the split, avoiding interference.
- Analysis: often a hypothesis test, plus confidence intervals and guardrail metrics.
- A decision: ship, don't ship, or test again, weighing the size of the effect and the cost.
The hypothesis test is step 5. Without the other steps, its p-value can be meaningless. A significant difference between this month and last month, for example, says nothing about what caused it.
Where people trip up
- Treating p < 0.05 as the decision. A significant 0.5% lift may not justify a costly change; a non-significant result doesn't prove the change is useless.
- Forgetting the design. No hypothesis test can rescue a broken split or a test stopped on a lucky day.
- Testing many things and reporting the winner. Each extra test is another chance for a false positive.
Bayesian A/B tests
Not every A/B test uses classical hypothesis testing. Bayesian analysis updates a prior belief with the data and reports, for instance, a 97% probability that B is better. There's no null to reject; the decision rule is based on probabilities and expected loss. The experiment design stays the same.
Our significance calculator shows the frequentist p-value and the Bayesian probability side by side. For a vocabulary-by-vocabulary comparison, see A/B testing vs hypothesis testing.
Common questions
Do all A/B tests use hypothesis testing?
Most use a frequentist hypothesis test such as a z-test. Bayesian A/B tests don't test a null hypothesis in the same way; they report probabilities such as the chance that B beats A and the expected loss of choosing it.
What is the null hypothesis in an A/B test?
Usually that the two versions have the same true conversion rate (or mean). The test asks whether the data is unusual enough under that assumption to reject it.