10,000 visitors each
- Control (A)
- 300 / 10,000 = 3.00%
- Variant (B)
- 360 / 10,000 = 3.60%
- Lift
- +20%
- Confidence
- 98.2% (p = 0.018)
Bayesian: about 99% chance B is better.
Free tool
Free A/B Test Calculator
Check statistical significance and plan your sample size. Plug in your numbers, get a clear answer.
Enter visitors and conversions for A and B. A two-proportion z-test and a Bayesian probability tell you whether your variant really won, or whether you're looking at noise. No account required.
Enter your test data
Control (A)
Variant (B)
What is an A/B test calculator?
An A/B test calculator tells you whether the difference between two versions of a page is real or just luck. Enter the visitors and conversions for your control (A) and your variant (B), and this free calculator works out each conversion rate, the lift, the statistical confidence and the probability that B is better.
Its Sample Size tab answers the question to ask before you start: how many visitors each version needs, and how many days that will take with your traffic.
How to use it
- Enter your data. Visitors and conversions for your control (A) and variant (B). The calculator handles the math in real time.
- Read the verdict. You get a clear winner/no-winner call at 95% confidence, plus the Bayesian probability that B beats A.
- Plan your next test. Switch to the Sample Size tab to figure out how much traffic you need before you start.
Statistical significance, explained simply
Every result gets two readings. They answer different questions, and together they tell you whether to ship.
The z-test: could this be chance?
A two-proportion z-test asks how surprising your difference would be if A and B really converted at the same rate. It is two-tailed, so it also catches a variant that is worse. When the confidence level reaches 95% (a p-value below 0.05), the result is statistically significant. Below that, you need more data.
Bayesian: how likely is B to win?
The Bayesian reading answers the question most people actually ask: what are the odds that B is better than A? The calculator estimates it by simulation (a Monte Carlo run over beta distributions). It is easy to explain, but it is not a stop sign: an early 85% can still be noise.
| Z-test (frequentist) | Bayesian | |
|---|---|---|
| Question it answers | Could this difference be chance? | How likely is B to beat A? |
| What you read | Confidence level and p-value | Probability that B wins |
| Calls a winner when | Confidence reaches 95% (p below 0.05) | No fixed line; read it as odds |
| Best for | A clear yes/no at the end of a test | Explaining the result to your team |
Question it answers
Z-test (frequentist)Could this difference be chance?
BayesianHow likely is B to beat A?
What you read
Z-test (frequentist)Confidence level and p-value
BayesianProbability that B wins
Calls a winner when
Z-test (frequentist)Confidence reaches 95% (p below 0.05)
BayesianNo fixed line; read it as odds
Best for
Z-test (frequentist)A clear yes/no at the end of a test
BayesianExplaining the result to your team
How much traffic do you need for an A/B test?
It depends on two things: your current conversion rate, and the smallest lift you want to be able to spot (the minimum detectable effect). Low conversion rates and small lifts need far more visitors. Here is what the calculator's Sample Size tab returns for common cases:
| Current conversion rate | To detect a 10% lift | To detect a 20% lift | To detect a 30% lift |
|---|---|---|---|
| 1% | 157,699 | 39,987 | 18,018 |
| 2% | 78,040 | 19,785 | 8,913 |
| 3% | 51,487 | 13,051 | 5,878 |
| 5% | 30,245 | 7,664 | 3,450 |
| 10% | 14,314 | 3,623 | 1,630 |
1%
To detect a 10% lift157,699
To detect a 20% lift39,987
To detect a 30% lift18,018
2%
To detect a 10% lift78,040
To detect a 20% lift19,785
To detect a 30% lift8,913
3%
To detect a 10% lift51,487
To detect a 20% lift13,051
To detect a 30% lift5,878
5%
To detect a 10% lift30,245
To detect a 20% lift7,664
To detect a 30% lift3,450
10%
To detect a 10% lift14,314
To detect a 20% lift3,623
To detect a 30% lift1,630
Visitors needed per variant (double it for the whole test), at 95% confidence, two-tailed, and 80% power: the settings the Sample Size tab uses (z = 1.96 and 0.8416). Lifts are relative: a 20% lift on 3% means 3.6%.
To turn that into a test length, divide the total for both variants by your daily visitors. Then round up to whole weeks, so every day of the week is counted.
Worked example: same lift, different verdict
Two tests with identical conversion rates. Only the amount of traffic differs.
2,000 visitors each
- Control (A)
- 60 / 2,000 = 3.00%
- Variant (B)
- 72 / 2,000 = 3.60%
- Lift
- +20%
- Confidence
- 71.2% (p = 0.288)
Bayesian: about 85% chance B is better.
Both tests show the same 20% lift (3.00% to 3.60%). With 10,000 visitors per variant the calculator is 98.2% confident, so B is a real winner. With 2,000 it is only 71.2% confident, well short of 95%, even though the Bayesian estimate already reads about 85%. The smaller test simply hasn't seen enough conversions to rule out luck.
The Sample Size tab would have warned you up front: at a 3% conversion rate, spotting a 20% lift needs 13,051 visitors per variant.
5 common A/B testing mistakes
- Stopping the moment it looks significant. Checking every day and stopping at the first 95% (peeking) makes false winners far more likely. Decide the sample size first and read the result once.
- Running for less than a full week. Weekday and weekend visitors behave differently. Run at least one full business cycle, usually one to two weeks, even if you hit your numbers sooner.
- Testing a change too small to detect. A 2% lift on a low-traffic page can take months to prove. Test bolder changes, or pick a minimum detectable effect your traffic can actually reach.
- Changing the test halfway through. Editing a variant or shifting the traffic split mid-test mixes two experiments into one result. Start a new test instead.
- Treating significant as important. With enough traffic, even a tiny lift becomes significant. Check the size of the lift and whether it is worth shipping.
Ready to run one? Leadpages has built-in A/B testing, and Smart Traffic can route each visitor to the version most likely to convert. For more on testing, read our guide to A/B testing landing pages, and write stronger variants with the free headline analyzer and CTA generator.
Frequently asked questions
Statistical significance means the difference between your variants is unlikely to be caused by chance. At 95% confidence, a difference this large would show up less than 5% of the time if A and B actually performed the same. This calculator uses a two-proportion z-test, which is the standard method for comparing two conversion rates.
It depends on your baseline conversion rate and how small a change you want to detect. A page converting at 3% needs roughly 13,000 visitors per variant to detect a 20% relative improvement. Lower conversion rates and smaller effects need more traffic. Use the Sample Size tab to get your exact number.
The frequentist result (confidence level, p-value) tells you whether to reject the hypothesis that A and B are the same. The Bayesian result tells you the probability that B is better than A. Both are valid. The Bayesian number is usually easier to act on: "there’s a 96% chance B wins" is more useful in a meeting than "p = 0.038."
Until you hit your required sample size, or at minimum one full business cycle (usually 1–2 weeks). Ending a test early because it "looks like a winner" is the most common A/B testing mistake. Early results are unreliable.
This calculator compares two variants (A vs B). If you’re running multivariate tests with three or more variants, you’d need to compare each pair separately and adjust for multiple comparisons. For most marketing tests, A/B (two variants) is the right approach.
Set it to the smallest improvement that would actually matter to your business. For most landing pages, 10–20% relative lift is a reasonable target. Going smaller (like 5%) means you need significantly more traffic and longer test durations.
It’s tempting, but no. Checking your results over and over and stopping the moment they cross 95% (called peeking) inflates your false-positive rate, so you end up shipping winners that aren’t real. Set your sample size before you start, run the test to that number, and read the result once at the end. The Bayesian probability is easier to follow along the way, but treat it as a progress check, not a reason to stop early.
Statistical significance tells you a difference is probably real. It does not tell you the difference is big enough to care about. With enough traffic, even a 0.1% lift can pass 95%. Look at the size of the lift and whether it is worth shipping and maintaining, not just whether you cleared the significance bar.
Use two-tailed unless you have a specific reason not to. A two-tailed test asks whether A and B differ at all, so it protects you when a variant performs worse than the control. A one-tailed test only asks whether B is better and needs less traffic, but it is blind to a variant that hurts. This calculator uses the more conservative two-tailed approach by default.
Get the A/B Testing Playbook
What to test first, how to run it, and the 6 mistakes that waste your traffic. One page, no fluff.
Next step
Ready to run your next test?
Build your variants in Leadpages and let built-in A/B testing and Smart Traffic find the winner.