Building blocks
- Normal distribution. The cumulative distribution uses Hart's double-precision algorithm as given by West (2005); the inverse uses Acklam's rational approximation refined with one Halley step. Both agree with SciPy to about 1e-12 in the range that matters.
- Gamma family. Log-gamma uses a Lanczos approximation. The chi-square tail probability uses the regularized incomplete gamma function (series or continued fraction).
Sample size (two proportions)
This is the pooled normal approximation without continuity correction, as in R's power.prop.test. With several variants, each group gets this n; the optional Bonferroni correction replaces α with α / (groups - 1).
Reference checks: p₁ = 0.20, p₂ = 0.25, α = 0.05, power 0.8 gives 1093.739 (R: 1093.739). p₁ = 0.10, p₂ = 0.12 gives 3840.847. One-sided p₁ = 0.10, p₂ = 0.12 gives 3025.315. p₁ = 0.05, p₂ = 0.06, α = 0.01, power 0.9 gives 15464.65.
Minimum detectable effect
Bisection on the sample size formula for the p₂ at which the required n equals your n, searched upward (lift) or downward (drop). Check: n = 1,000, p₁ = 0.20 gives p₂ = 0.25239, matching a root-finder in SciPy.
Duration
Days = ceil(n per group × groups / (daily visitors × share in test)), also shown rounded up to whole weeks with a minimum of seven days.
Significance (frequentist)
Checks: 200/1000 vs 250/1000 gives z = 2.6774, p = 0.0074197, χ² = 7.1685 (SciPy chi2_contingency, correction off, gives the same). 320/10000 vs 380/10000 gives p = 0.020969.
Bayesian probability to beat
Posteriors Beta(1 + x, 1 + n - x) for each group. P(B > A) uses the exact closed-form sum described by Evan Miller:
For extremely large counts (more than 200,000 conversions in B) it switches to a normal approximation of the two Beta distributions. Credible intervals and expected loss use 40,000 seeded Monte Carlo draws (Marsaglia-Tsang gamma sampler), so output is reproducible.
Checks against numerical integration: 200/1000 vs 250/1000 gives 0.996283; 32/1000 vs 41/1000 gives 0.856845; Beta(4,8) vs Beta(6,6) gives 0.806502; identical data gives 0.5.
Sample ratio mismatch
Chi-square goodness of fit: χ² = Σ (observed - expected)² / expected with groups - 1 degrees of freedom. Default alarm threshold p < 0.001.
Checks (SciPy chisquare): 1000 vs 1100 gives χ² = 4.7619, p = 0.029096. 50000 vs 50600 gives p = 0.058531. 9100 vs 900 on a 90/10 plan gives p = 0.00085812.
Price tests
Revenue per visitor = CR × units per order × price. Gross profit per visitor = CR × units per order × (price - unit cost). Break-even conversion for the new price = CR_A × (P_A - C) / (P_B - C). The significance test applies to conversion only.
A/A simulation
Each simulated test draws binomial counts for two identical groups (exact geometric-skip sampling for small expected counts, normal approximation otherwise), checks a two-sided z-test at each look, and records significance at the final look and at any look. Checks: with one look the false-positive rate is within ±1.2 points of 5% over 4,000 runs; with ten looks the stop-at-first-significance rate is within ±2.5 points of the 19.3% reported by Armitage, McPherson and Rowe.
X post tools
The post comparer treats each impression as a trial and each engagement as a success (capped at impressions) and runs the same z-test and Bayesian calculation. The character counter follows X's published weighting: most Latin-script characters count 1, other characters and emoji count 2, and links count 23.
Known limitations
- Normal approximations get rough with very small counts (fewer than about 5 expected conversions per group). Treat those results as indicative.
- The tools assume each visitor is counted once and assigned independently. Per-session or per-pageview counting breaks that assumption.
- The significance calculator is a fixed-horizon test. It isn't valid if you checked repeatedly and stopped on a good result.
- Revenue and profit significance need order-level data and aren't computed.
The source is readable in your browser at /assets/abstats.js.