How the duration is calculated
The calculator first works out the sample size per group, using the same formula as the sample size calculator. Then:
With the defaults (2,500 visitors a day, 2.5% baseline, 15% relative lift, 95% confidence, 80% power) you need 29,193 visitors per group, 58,386 total, which takes 24 days. Round to 28.
Why whole weeks
Shopping behavior changes across the week. Weekend visitors browse more on phones; weekday lunchtime visitors may convert differently from Sunday night visitors. If a test runs Monday to Thursday, it's measuring a skewed mix. Running whole weeks keeps the mix the same as your normal traffic.
Watch out for unusual weeks too. A test that spans Black Friday, a big email campaign, or a stockout is measuring that event as much as your change.
When the number is too big
If the calculator says 90 days, the test isn't practical. Your options:
- Test a bigger change. Doubling the MDE cuts the sample roughly by four.
- Test higher in the funnel. Add-to-cart happens more often than purchase, so it needs fewer visitors. Just confirm the change doesn't hurt purchases later.
- Pool pages. Test a template change across all product pages instead of one product.
- Accept more risk. A 10% significance level (90% confidence) needs about a fifth less traffic. Be honest about that choice when you report results.
The MDE calculator works this backwards: give it the time you have and it tells you the smallest effect you can find.
Common questions
How long should an A/B test run?
Until it reaches the sample size you planned, and for at least one full week, ideally two. For a page with 2,500 daily visitors, a 2.5% baseline and a 15% target lift, that's 24 days, so plan for 28. Ending on a full week means every weekday is equally represented.
Can I stop a test early if it's clearly winning?
Not with a standard fixed-sample test. Stopping the first time p dips below 0.05 inflates false positives. With 10 checks the real false-positive rate is around 19%, not 5%. If you need to stop early, use a method designed for it, such as a sequential test with planned interim looks.
What's the longest a test should run?
There's no fixed rule, but past six to eight weeks the problems pile up. Cookies expire and returning visitors get re-bucketed, seasonality shifts the baseline, and other site changes interfere. If a test needs that long, the effect you're chasing is probably too small for your traffic.
Does running a test on only part of my traffic make sense?
It's useful when a change is risky, because fewer people see a broken version. The cost is time. Sending 50% of traffic into the test doubles the duration. Set the share field to see the effect.