How the comparison works
Each impression counts as one trial and each engagement as one success. The comparer then runs the same two tests as the significance calculator:
- a two-sided two-proportion z-test for the engagement rates, and
- the exact Bayesian probability that post B's true rate is higher than A's (flat Beta(1,1) priors).
In the default example, post A has 331 engagements on 18,400 impressions (1.80%) and post B has 383 on 16,950 (2.26%). B is up about 25.6%, p = 0.002, and the chance that B's true rate is higher is 99.9%.
The big caveat
In a real A/B test, the only difference between groups is the change. Two posts differ in much more: time of day, the news cycle, who happened to be online, and how X's ranking chose to distribute each one. A significant result says the gap isn't random noise within those two posts. It doesn't prove the wording caused it.
You can get closer to a fair test by:
- posting both variants in the same time slot on comparable days,
- alternating which one goes first (the variant planner builds an AB/BA schedule),
- running several pairs and adding up the totals before comparing,
- changing one thing at a time.
Read the full method in how to A/B test posts on X.
ABFinal isn't affiliated with X Corp. The comparer doesn't connect to your account; you type the numbers in.
Common questions
How do I calculate engagement rate on X?
Divide engagements by impressions. A post with 383 engagements on 16,950 impressions has an engagement rate of 2.26%. X's analytics shows both numbers for each post.
Can I A/B test tweets?
Not in the strict sense for organic posts. X doesn't randomly split your followers between two versions of a post. You can compare posts, and paid campaigns in X Ads let you run variants against the same audience. For organic posts, repeat the comparison across several pairs to reduce the effect of timing.
What counts as an engagement?
X's analytics counts interactions such as likes, replies, reposts, link and profile clicks, and media views and expands. Exactly what's included can change, so compare posts using the same metric from the same screen.
Why does the comparer warn about engagements exceeding impressions?
One person can engage several times with one post, for example by liking it and clicking the link. On small posts that can push engagements above impressions. The test treats each impression as one trial, so it caps engagements at impressions and warns you.