🧪 A/B Test Significance Calculator
An AB test calculator for significance: p-value, Z-score, the confidence interval on the difference, observed power, and the sample size to plan.
A difference this large would occur by chance about 9.1% of the time even if the variants were identical — too often to call a winner. Collect more data or accept a lower confidence level.
What A/B Test Significance Calculator Does
An A/B test compares two conversion rates measured on samples, and asks whether the gap between them is bigger than random variation would comfortably produce. It cannot prove that B is better than A. It can only tell you how surprising your data would be if B and A were in truth identical — which is what a p-value is.
That distinction sounds pedantic and is the source of nearly every bad decision made from test results. A p-value of 0.04 does not mean there is a 96% chance the variant wins. It means that if the variants were identical, you would see a difference this large about 4% of the time anyway.
This calculator uses a two-proportion Z-test, the standard frequentist method. It reports every intermediate value — both standard errors, the standard error of the difference, the Z-score, the p-value, observed power and the confidence interval on the difference — so you can check the arithmetic rather than trust a verdict. It also computes the sample size you should fix before the test starts.
How to Use A/B Test Significance Calculator
- Choose "Evaluate a finished test" to analyze results, or "Plan sample size" before you start
- Enter visitors and conversions for the control (A) and the variant (B)
- Pick your confidence level — 95% is the standard choice
- Choose a two-sided hypothesis unless you have a specific reason to test one direction only
- Read the p-value and confidence interval, not just the significant/not-significant verdict
Formula Used by A/B Test Significance Calculator
Two-proportion Z-test
Z = (p̂_B − p̂_A) ÷ √( SE_A² + SE_B² )
- p̂_A, p̂_B
- Observed conversion rates: conversions ÷ visitors in each group
- SE_A
- √( p̂_A(1 − p̂_A) ÷ n_A ) — standard error of the control rate
- SE_B
- √( p̂_B(1 − p̂_B) ÷ n_B ) — standard error of the variant rate
- Z
- How many standard errors apart the two rates are
Worked example
Control: 80,000 visitors, 1,600 conversions. Variant: 80,000 visitors, 1,696 conversions.
- p̂_A = 1600 ÷ 80000 = 0.0200 (2.00%)
- p̂_B = 1696 ÷ 80000 = 0.0212 (2.12%)
- SE_A = √(0.02 × 0.98 ÷ 80000) = 0.000495
- SE_B = √(0.0212 × 0.9788 ÷ 80000) = 0.000509
- SE_diff = √(0.000495² + 0.000509²) = 0.000710
- Z = (0.0212 − 0.0200) ÷ 0.000710 = 1.6897
Result: Z = 1.69. Two-sided p = 0.0911 — not significant at 95%. One-sided p = 0.0455 — significant at 95%. The same data, two answers, decided entirely by a choice you make before looking.
Confidence interval on the difference
CI = (p̂_B − p̂_A) ± z_crit × SE_diff
- z_crit
- 1.96 for a two-sided 95% interval; 1.645 for one-sided 95%
- CI
- Range of absolute differences in percentage points consistent with the data
Worked example
The same test: difference 0.12 percentage points, SE_diff 0.000710.
- Margin: 1.96 × 0.000710 = 0.001392
- Lower: 0.0012 − 0.001392 = −0.000192
- Upper: 0.0012 + 0.001392 = 0.002592
Result: −0.02 to +0.26 percentage points. The interval crosses zero, which is the same conclusion as p > 0.05 — but it also shows the plausible upside is small. This number is more useful than the verdict, and no competing calculator reports it.
Required sample size per variant
n = [ z_α√(2p̄(1−p̄)) + z_β√(p₁(1−p₁) + p₂(1−p₂)) ]² ÷ (p₂ − p₁)²
- p₁
- Baseline conversion rate
- p₂
- Rate you want to be able to detect: p₁ × (1 + minimum detectable lift)
- p̄
- Average of p₁ and p₂
- z_α
- Critical value for your confidence level (1.96 two-sided at 95%)
- z_β
- Critical value for power (0.8416 at 80% power)
Worked example
Baseline 2%, detect a 10% relative lift (2.00% → 2.20%), 95% confidence two-sided, 80% power.
- p₁ = 0.02, p₂ = 0.022, p̄ = 0.021
- z_α = 1.9600, z_β = 0.8416
- Numerator: [1.96×√(2×0.021×0.979) + 0.8416×√(0.0196 + 0.021516)]² = 0.322726
- Denominator: (0.022 − 0.02)² = 0.000004
Result: About 80,700 visitors per variant, 161,400 total. Small lifts on low base rates are expensive to prove — which is why most tests that "failed" were simply never large enough.
Sample Size Needed per Variant (95% confidence, 80% power, two-sided)
Required traffic rises roughly with the square of how small the effect is. Halving the lift you want to detect quadruples the traffic you need.
| Baseline rate | +5% lift | +10% lift | +20% lift | +50% lift |
|---|---|---|---|---|
| 1% | 637,010 | 163,095 | 42,693 | 7,750 |
| 2% | 315,206 | 80,682 | 21,109 | 3,826 |
| 5% | 122,124 | 31,234 | 8,158 | 1,471 |
| 10% | 57,763 | 14,751 | 3,841 | 686 |
| 20% | 25,583 | 6,510 | 1,683 | 294 |
What the p-value Actually Says
The most common misreadings, side by side with the correct statement.
| p-value | Correct reading | Common misreading |
|---|---|---|
| 0.04 | If A and B were identical, data this extreme would appear 4% of the time | "There is a 96% chance B is better" |
| 0.20 | The data are consistent with no difference; you have not learned much | "B is 80% likely to win" |
| 0.001 | Very surprising under the null hypothesis | "The effect is large and important" |
| 0.06 | Just short of the threshold you chose in advance | "Almost significant" — run it longer |
Validity Checks Before You Trust Any Result
A statistically significant result from an invalid test is still wrong. These are the checks that the calculator cannot do for you.
| Check | What to look for | Why it matters |
|---|---|---|
| Sample ratio mismatch | Traffic split materially off from the intended ratio | Indicates a bug in assignment or tracking; invalidates the whole test |
| Full business cycles | At least 1–2 complete weeks | Weekday and weekend visitors convert differently |
| Fixed stopping point | Sample size decided before the test began | Peeking inflates false positives far above your stated α |
| Novelty effect | Early lift that decays over time | Returning users react to change itself, not to the change being better |
| Multiple variants | More than one variant tested at once | Each extra comparison adds another chance of a false positive |
How to Read Your Result
Read the confidence interval, not the verdict
Significant or not is one bit of information thrown away from a much richer number. An interval of +0.1% to +4.0% and an interval of +1.9% to +2.1% can both be "significant", but they justify completely different decisions. If the interval spans from "barely worth shipping" to "transformative", you have not finished measuring.
The peeking problem
Checking the dashboard daily and stopping the moment p drops below 0.05 does not give you a 5% false positive rate. Every look is another chance for random noise to cross the line, and with enough looks a null test will eventually appear significant. Fix the sample size in advance and read the result once. If you genuinely need to monitor continuously, you need a sequential testing method designed for it, not a fixed-horizon test checked repeatedly.
One-sided tests are not free
A one-sided test needs a smaller Z to reach significance, which is exactly why it is tempting. The example above is significant one-sided and not significant two-sided on identical data. The cost is that a one-sided test cannot tell you B is worse — it treats any negative result as "no effect". Choose the direction before you see the data, or use two-sided.
Underpowered wins overstate the effect
When power is low, only unusually large observed differences clear the significance threshold. So among the tests that do reach significance, the measured lift is systematically inflated — sometimes by a factor of two or more. This is why a 40% lift from a small test rarely survives being shipped. The direction may be right; the magnitude usually is not.
Limitations & Accuracy Notes
- This is a fixed-horizon frequentist test. It assumes you chose the sample size in advance and evaluated once. Applying it to a test you monitored continuously does not give the error rate it reports.
- It compares exactly two groups on a single binary metric. Testing several variants at once multiplies the false-positive rate, and correcting for that (Bonferroni or similar) requires a stricter threshold than the one selected here.
- The normal approximation behind the Z-test degrades when conversion counts are very small. As a rule of thumb it needs at least about 5 conversions and 5 non-conversions in each group; below that, use an exact test such as Fisher's.
- Conversion rate is not revenue. A variant can lift conversions while lowering average order value, and a binary test will never show it. Test the metric you actually care about.
- Statistical significance says nothing about business significance. A real, reproducible 0.05% lift may be entirely not worth the engineering cost of shipping it.
- The calculator cannot detect sample ratio mismatch, tracking bugs, or contamination between groups. Those are the failures that most often invalidate real tests, and they have to be checked separately.
Frequently Asked Questions
What does statistical significance actually mean in an A/B test?
Should I use a one-sided or two-sided test?
Why should I calculate sample size before running the test?
My test is significant but power is low. What does that mean?
How long should an A/B test run?
What does statistical significance actually mean?
Why should I not check the test repeatedly?
How large a sample do I need?
My variant is up 15% — is that real?
What is a false discovery when testing many variants?
Is my test data stored?
References & Further Reading
- NIST/SEMATECH e-Handbook — Comparing two proportions — US federal reference for the two-proportion test and its sample size formula