A/B test calculator
Plan the test before you run it, and read the result after. Sample size, duration, significance and a sample-ratio check, all on one page.
Sample size uses a two-proportion test with a pooled null. Run at least one full business cycle so weekday and weekend traffic both count, even when the number here is reached sooner.
At 95% confidence, variant B’s rate of 6.00% differs from control’s 5.00% by more than chance would explain (+20.00% relative).
The 4,000 / 4,000 split is consistent with an even 50/50 assignment (χ² 0.0, p 1.0000).
The terms behind the two calculators, in plain words. Open any one for the longer version and a worked example.
It is conversions divided by visitors, for one arm. Every other number on this page is built from two of these rates and the counts behind them. A rate on its own says nothing about whether a gap is significant, only how big it looks.
240 sign-ups from 4,000 visitors is a 6.00% conversion rate.
Set before the test, not after. A smaller MDE needs far more traffic, because separating a tiny real change from noise takes more evidence. Relative MDE is a share of the baseline; absolute MDE is in percentage points. Pick the smallest change you would ship.
A 10% relative MDE on a 5% baseline means you want to catch a move to 5.5% or better.
Power is one minus the false-negative rate. At 80% power, a test that truly has an effect of your MDE will reach significance four times out of five. Raising power to 90% catches more real winners but costs more sample.
80% power means one real winner in five still slips through as a null.
The p-value is the probability of seeing a gap this large, or larger, if the two arms were really the same. A small p-value means the observed gap would be a surprise under no effect. Confidence is one minus your p-value threshold: 95% confidence tests against p below 0.05.
p = 0.02 says a gap this big would happen about twice in a hundred repeats of a null test.
Significant means the p-value fell below your threshold, so you reject the idea that the arms are identical. It does not measure how big or how valuable the effect is, and it is not proof. A significant loss is still a loss. Read it next to the uplift and the confidence interval, never alone.
At 95%, p = 0.03 is significant; p = 0.08 is not, however promising the uplift looks.
A two-sided test asks whether the arms differ at all, in either direction. A one-sided test asks only whether the variant is better, and reaches significance sooner for the same data. Use two-sided unless you have a firm reason to ignore the chance the variant is worse.
A one-sided p-value is half its two-sided twin, so it is easier to reach.
Instead of a single uplift number, the interval gives a band. A 95% interval that sits entirely above zero is another way of seeing a significant win. An interval that straddles zero tells you the true effect might be nothing, or might even be negative.
An uplift of +0.4pp with a 95% interval of −0.1pp to +0.9pp still includes zero.
You meant to send half the traffic to each arm. SRM checks whether the counts that actually landed match that plan. A mismatch usually points to a bug in bucketing, a redirect that drops one arm, or a bot filter hitting one side. When it fires, fix the plumbing before you read any verdict.
A 52/48 split on 10,000 visitors is unlikely by chance and would flag as a mismatch.
All the maths runs in your browser. Your numbers never leave this page.
Plan first, then analyse
Size the test before you start, then bring the counts back to read whether the difference is significant.
Enter your baseline rate, the smallest lift worth catching, and your traffic. You get the sample each arm needs and how long that takes to reach.
Drop in visitors and conversions for control and variant. You get the uplift, the p-value, whether it is significant, a confidence interval and a sample-ratio check.
Power, MDE, p-values and significance in plain words, each with a worked example.
Questions
- Do my numbers get sent anywhere?
- No. Every calculation runs in your browser with client-side JavaScript. Nothing you type is uploaded, logged or stored.
- Which significance test does it use?
- A two-proportion z-test with a pooled null for the p-value, and an unpooled standard error for the confidence interval on the absolute difference. Sample size uses the same two-proportion basis.
- What is a sample ratio mismatch check?
- A chi-square test on the visitor counts against your intended split. If the split that landed is too far from even, it flags a likely bucketing bug before you trust the verdict.
- Should I use one-sided or two-sided?
- Two-sided unless you have a firm reason to ignore the chance the variant is worse. A one-sided test reaches significance sooner but only answers whether the variant is better.
More on how optimiser.pro reads a test on the how it works section, or see the workspace preview.
Run it on your own experiments
This page stops at whether the rate moved. Optimiser.pro carries it through to the payment data: which arm earned more per acquired customer, not only which one converted.