Why a post comparison isn't a true A/B test
In a website A/B test, each visitor is randomly assigned to A or B at the same moment, so the only systematic difference between the groups is the change. With posts, A and B go out at different times. They meet different news, different people online, and different decisions from X's ranking system. A single pair can differ by a factor of three for reasons that have nothing to do with the words.
So the goal is not to prove causation from one pair. It's to collect enough matched pairs that timing noise averages out and the real pattern shows.
Step 1: Pick one variable
Change one thing and keep everything else as similar as you can. Good candidates:
- the first line (hook),
- with or without an image,
- short against long,
- the call to action,
- link in the post or in the first reply,
- single post or thread,
- posting time.
If you change the hook and add an image, you won't know which one did the work.
Step 2: Choose your metric and baseline
Engagement rate (engagements divided by impressions) is shown for every post in X's analytics. Find your median over your last 20 to 30 posts. That's your baseline.
Step 3: Work out how much data you need
Use the variant planner. At a 1.8% baseline, detecting a 25% relative improvement takes about 15,379 impressions per variant. An account averaging 4,000 impressions a post needs four posts of each style. An account averaging 800 would need twenty, which tells you to test something bolder.
Step 4: Post in matched pairs
For each round, write two new posts on a similar topic: one in style A, one in style B. Post them in the same time slot on similar days. Alternate which goes first in an AB, BA, BA, AB pattern so that neither style always gets the Monday slot.
Step 5: Log the numbers at a fixed age
Record impressions and engagements for every post at the same age, for example 48 hours after posting. Posts keep collecting impressions for a while, and comparing a two-day-old post with a two-hour-old one is unfair.
A simple sheet works: date, slot, variant, impressions, engagements.
Step 6: Compare the totals
When the rounds are done, add up impressions and engagements for all A posts and all B posts. Enter the totals in the post A/B comparer. It reports the engagement rates, the relative difference, a p-value and the probability that B's true rate is higher.
If the result is significant and consistent across most pairs, you've found something worth adopting. If one pair carries the whole result, be suspicious: that's often a single post that took off for reasons unrelated to the style.
Step 7: Decide and move on
Adopt the winner as your default and test the next idea against it. A small library of tested defaults is worth more than a long list of untested hunches.
ABFinal isn't affiliated with X Corp. Check X's own help pages for current analytics definitions.
Common questions
How long should a tweet A/B test take?
Long enough to collect the impressions your plan calls for, usually two to six weeks for a mid-sized account posting a few test pairs each week. Smaller accounts need either more time or bolder differences between variants.
Should I delete and repost a tweet to test it?
No. Reposting the same text is a weak test because the second version reaches people who already saw the first. Write fresh pairs that share the variable you're testing.
Is engagement rate the best metric?
It's the most widely available. If your goal is traffic, use link clicks per impression; if it's growth, use follows per impression. Choose before you start and stick to it.
Can I use paid ads to test tweet copy?
Yes. X Ads lets you run several creatives in the same campaign, which gets you closer to a real randomized test because the ads compete for the same audience at the same time. It costs money, but the answers come faster and cleaner.