ABFinal

Running tests well

How to A/B Test Email Subject Lines

Short answer

Split a random part of your list into equal groups, send each group one subject line, judge the result on clicks or orders rather than opens, and only call a winner when the sample was big enough to detect the difference you care about.

Key takeaways

  • Open rate has been unreliable since Apple Mail Privacy Protection began loading tracking pixels automatically in 2021.
  • Clicks or orders per delivered email are better success metrics, but they need far larger samples.
  • At a 3% click rate you need roughly 14,000 recipients per subject line to detect a 20% relative lift.
  • Small lists can still learn by repeating the same kind of test across several sends.

Subject line tests look easy because email tools make them easy. Write two versions, pick a test percentage, press send, and the tool announces a winner a few hours later. The trouble is that many of those winners would not win again. Here's how to set up a test whose result you can act on.

Why is open rate a weak metric for subject line tests?

Open rate is unreliable because Apple Mail Privacy Protection preloads email images, which records opens that never happened.

Opens are tracked with a tiny image. When it loads, the email counts as opened. Since Apple introduced Mail Privacy Protection in 2021, Apple Mail can download that image in the background for users who turn the feature on, whether or not they read the message. Those subscribers show up as openers of nearly everything.

That does two things to a test. It inflates both groups by the same crowd of automatic opens, which shrinks the visible difference. And it adds noise, because how many Apple Mail users land in each group varies by chance. A subject line can "win" on opens purely because more pixel-loading readers ended up in its half.

Opens still have uses, such as spotting a deliverability problem. For choosing between subject lines, use something people have to do on purpose.

Which metric should decide the winner?

Clicks per delivered email are the best default, and orders or revenue per delivered email are better still when you have the volume.

Metric Measures Typical rate Sample needed
Opens Pixel loads, partly automatic High Small, but biased
Clicks per delivered A real decision to engage Low Large
Orders per delivered The business result Very low Very large

Use "per delivered", not "click-to-open rate". Click-to-open divides by opens, so it inherits the same problem.

How many recipients do you need?

For a 3% click rate and a 20% relative lift, you need roughly 14,000 delivered emails per subject line at 95% confidence and 80% power.

That figure comes from the standard two-proportion formula, and you can check it in the sample size calculator by entering a 3% baseline and a 20% relative effect. Two subject lines therefore need about 28,000 recipients before the rest of the list gets the winner.

Many lists are smaller than that. Three options:

  1. Test bigger differences. Tiny wording changes rarely move clicks by 20%. A question versus a statement, or a number versus no number, is more likely to.
  2. Send the test to the whole list as a straight 50/50 split instead of a small test cell. You lose the "send the winner to the rest" step, but you gain power.
  3. Repeat the same kind of test over several campaigns and pool the results, for example "subject lines with a price" against "subject lines without".

How do you set up the split?

Randomly assign recipients to equal groups, send both versions at the same time and change nothing except the subject line.

  1. Write one hypothesis, such as "naming the product in the subject line raises clicks".
  2. Write two subject lines that differ only on that idea. Keep the preheader, sender name and content identical.
  3. Let the email tool randomise who gets what. Never split by signup date, domain or alphabet.
  4. Send both versions at the same moment. A two-hour gap turns the test into a send-time test.
  5. Decide the waiting time and the metric before you send, and write them down.

When is the winner real?

Treat a result as a winner only when the planned sample was reached and the difference is significant on the metric you chose in advance.

Suppose version A went to 14,000 people and got 420 clicks (3.0%), and version B went to 14,000 and got 500 clicks (3.6%). Put those counts into the significance calculator. The difference is significant at the 5% level, with a confidence interval for the lift that stays above zero. That is a result you can use.

Now suppose the test cell was 2,000 per version and the counts were 60 against 70 clicks. B looks 17% better, but the p-value is far above 0.05. Sending B to the rest of the list is harmless, but don't record it as a lesson about what your audience prefers.

Two more checks before you trust a winner:

  • Did both versions actually reach about the same number of inboxes? A big gap in delivered counts can point to a spam filter treating one subject line differently.
  • Did unsubscribes or spam complaints rise for the winner? A curiosity-bait subject line can buy clicks and cost subscribers.

What should you record after each test?

Write down the hypothesis, both subject lines, counts, the metric, the result and one sentence on what you learned.

Single tests are noisy. A log of twenty tests is where patterns show up, such as "subject lines with the product name win most of the time for promotional sends but not for newsletters". The guide on how to analyze A/B test results has a fuller checklist you can adapt.

Common questions

How big should the test group be for a subject line test?

Big enough for the effect you want to find. Work it out with a sample size calculator using your usual click rate. If each subject line would need more recipients than you have, test bolder differences or pool results across several campaigns.

How long should I wait before picking the winning subject line?

Most email clicks arrive in the first day, but wait at least 24 hours if you can, and longer if your audience reads email mainly on weekdays. Picking a winner after two hours favours people who open email instantly, who may not behave like the rest of your list.

Is it OK to judge a subject line test on open rate?

Only with care. Apple Mail Privacy Protection preloads images, which can mark emails as opened even when no one read them. If many of your subscribers use Apple Mail, opens will be inflated and noisy, so use clicks or conversions to decide.

Can I test more than two subject lines at once?

Yes, but every extra version needs its own full sample and raises the chance of a false winner. Two versions are usually the better choice unless your list is very large.

Sources

  1. Apple Support: Use Mail Privacy Protection on iPhone
  2. ABFinal methodology: formulas and reference checks