Short answer
CUPED (Controlled-experiment Using Pre-Experiment Data) adjusts each user's test metric by how they behaved before the test started, which removes noise that has nothing to do with the change; with a pre-period correlation of 0.5 it cuts the required sample by about a quarter.
Key takeaways
- CUPED was published by Microsoft researchers working on Bing experiments in 2013.
- It subtracts the part of each user's metric that their pre-test behaviour already predicts.
- The variance falls by a factor of one minus the squared correlation between pre-test and test data.
- It helps most for returning users and continuous metrics, and barely at all for first-time visitors.
Most of the noise in an A/B test comes from people being different from each other. A heavy user spends ten times what a light user spends, whatever version of the page they see. CUPED is a way of taking that known difference out of the comparison, so the effect of your change stands out with less data.
What does CUPED stand for?
CUPED stands for Controlled-experiment Using Pre-Experiment Data, a variance reduction method described by Alex Deng, Ya Xu, Ron Kohavi and Toby Walker in 2013.
The authors worked on Microsoft's experimentation platform, where Bing ran large numbers of tests and wanted answers sooner. Their paper showed that using each user's metric from the weeks before a test as a covariate made many experiments far more sensitive. It reports a variance reduction of about half on an important Bing metric, which is the same as doubling the traffic.
How does CUPED work?
For each user, CUPED subtracts the part of their test-period metric that their pre-test metric predicts, then compares the adjusted values between groups.
The adjusted metric is:
Y_adj = Y − θ × (X − mean of X)
- Y is the user's metric during the test, for example revenue.
- X is the same metric for the same user before the test started.
- θ is the covariance of X and Y divided by the variance of X, estimated from all users together.
A user who spent a lot before the test gets their test spending nudged down; a light user gets nudged up. Their differences in habit are removed, and what remains is closer to the effect of the version they saw. Because X was measured before anyone was randomised, the groups have the same X on average, so the adjustment does not bias the comparison.
How much sample can CUPED save?
The variance falls by a factor of 1 − ρ², where ρ is the correlation between pre-test and test-period values, and the sample you need falls by the same factor.
| Correlation (ρ) | Variance kept | Sample saved |
|---|---|---|
| 0.3 | 91% | about 9% |
| 0.5 | 75% | about 25% |
| 0.7 | 51% | about 49% |
| 0.9 | 19% | about 81% |
Take a test that the sample size calculator says needs 40,000 users per group. If your metric has a pre-period correlation of 0.7, CUPED brings that down to roughly 20,400 per group. You can run the test in about half the time, or keep the time and detect a smaller effect. To see what a smaller detectable effect is worth to you, try the MDE calculator with the same traffic.
A small worked example shows the mechanics. Two users are in the treatment group. Before the test, user 1 spent $200 a month and user 2 spent $20; the average across all users was $60. During the test they spent $210 and $35. With θ = 0.8, user 1's adjusted value is 210 − 0.8 × (200 − 60) = $98, and user 2's is 35 − 0.8 × (20 − 60) = $67. The raw gap of $175 between them shrinks to $31. Repeat that across thousands of users and the spread of the metric drops sharply, while the average difference between groups stays where it was.
When does CUPED work well?
CUPED works best when the same users return often and the metric is stable for each user, such as sessions, minutes or spend per user.
Good fits:
- Subscription products and apps where most users were active before the test.
- Engagement metrics such as searches, page views or time spent.
- Revenue per user on a site with many repeat buyers.
Weak fits:
- First-time visitors. They have no pre-period data, so their X is missing or zero and the adjustment does little for them.
- Rare binary outcomes, such as a single purchase by a new visitor. Past behaviour predicts them poorly.
- Very short pre-periods, which give noisy X values and a weak correlation.
For a typical small online store, where most buyers arrive once, the gain on conversion rate is often modest. On revenue from returning customers it can be large.
What can go wrong?
The main risks are using data the treatment could have influenced and choosing the covariate after seeing the results.
- The pre-period must end before the test begins. If it overlaps, the change itself leaks into X.
- Decide on CUPED and the covariate before the test. Trying several adjustments and keeping the best-looking one is a form of p-hacking.
- Handle users with no pre-period data consistently, for example by setting X to the overall mean so their adjustment is zero.
- Check the split as usual. CUPED does not fix a sample ratio mismatch or a broken assignment.
Do you need special software for CUPED?
No; it is a few lines in SQL, Python or R, and several commercial testing tools offer it as a setting.
The steps: pull each user's pre-period metric, estimate θ across all users, compute the adjusted metric, then run your usual t-test on the adjusted values. The standard error shrinks, the point estimate stays put. If you already export per-user data, it fits into your current analysis.
Common questions
Does CUPED change the size of the measured effect?
No. Because the pre-test data was collected before randomisation, it is balanced between the groups on average, and the adjustment leaves the expected difference unchanged. It narrows the confidence interval around that difference.
Can I use CUPED on conversion rate?
You can apply it to a 0/1 conversion metric, but the gain depends on how well past behaviour predicts it. For stores where most buyers are new visitors, the correlation is low and the variance reduction is small.
What pre-experiment period should I use?
Use a window long enough to capture normal behaviour, often one to several weeks before the test starts, and make sure it ends before anyone saw the new version. Data from after the test began could be affected by the change and would bias the result.
Is CUPED the same as regression adjustment?
It is a simple form of it. CUPED adjusts with one pre-period covariate; regression adjustment can use several. Both rely on covariates that the treatment could not have affected.