Statistical Significance Calculator
Enter raw visitors and conversions from any A/B test and get the verdict the math supports — p-value, confidence interval and an honest read. Then size the next test properly before it starts. Free, no signup, and your numbers never leave the browser.
Did your variant actually win?
A two-sided z-test for two proportions — the same math serious testing platforms run under the hood. Everything computes locally in your browser; nothing is uploaded anywhere.
Enter your test numbers
Fill in visitors and conversions for both groups and the verdict updates instantly.
Method: two-proportion z-test with pooled standard error; the confidence interval on the difference uses unpooled standard errors. A significant p-value is necessary but not sufficient — the test also has to be sized and run cleanly. See the rules below.
How much traffic does a trustworthy test need?
Undersized tests produce coin-flip verdicts. Size the test before it starts, then run it to completion — even when an early peek looks exciting.
Assumes an even 50/50 split, two-sided testing at the confidence level selected in the calculator above, and that the lift is relative — detecting a 15% lift on a 3% baseline means reaching 3.45%. Whatever the math says, never run less than two full business weeks: weekday and weekend visitors are different populations.
Evidence, not opinion
Most site work is judged by the wrong number. Being precise about measurement is the foundation everything else stands on.
Conversion rate is orders divided by sessions — a blended average across every visitor a site receives. It is not a property of the site alone. It moves with traffic mix, channel strategy, seasonality, pricing and promotions, whether or not a single line of code changes. Returning email subscribers convert at multiples of cold social traffic; branded search converts far above prospecting video ads; a newly launched international market converts below the home market while it finds its footing.
That means the blended number can move in the opposite direction of the business:
A concrete illustration
Imagine a store whose only traffic is Facebook, converting at 3%. Site performance work then makes YouTube ads viable — a colder, cheaper audience that converts at 1%:
| Scenario | Sessions | Per-channel conversion | Orders | Blended conversion |
|---|---|---|---|---|
| Before — Facebook only | 100,000 | 3.0% | 3,000 | 3.0% |
| After — performance unlocks YouTube | 100,000 + 50,000 | 3.0% / 1.0% | 3,500 | 2.3% |
| Business result | +50% | Unchanged in each channel | +17% | −22% |
Blended conversion "fell" from 3.0% to 2.3% — while orders grew 17% and revenue grew with them. Nothing broke: the site got better, which is what made the colder channel viable at all. The reverse trap is just as real: cut prospecting spend and retarget only warm audiences, and blended conversion rises while the business shrinks and the site quietly decays.
On its own, conversion rate grades the traffic, not the site. A falling blended rate can be the signature of successful growth — and a rising one can mask decay.
The clean instrument is a controlled A/B test: the same traffic, over the same period, randomly split, with the site change as the only difference between the groups. Traffic mix, seasonality and campaign changes hit both groups equally and cancel out — what remains is the site's causal contribution.
That's why revenue-relevant changes we ship go out as controlled experiments — and why the right question when evaluating site work is never "what did the blended rate do this month?" but "what did the tests show?"
A significant p-value is necessary, not sufficient
The calculator's verdict only holds if the test was run cleanly. These are the rules we run experiments by — break them and the p-value stops meaning what you think it means.
Size it first, then don't peek
Checking results early and stopping at the first significant reading inflates the false-positive rate from 5% to over 25%. Decide the sample size up front with the planner above; read the verdict once, at the end.
Run full weeks
Weekday and weekend visitors behave differently, and paydays bend the curve. Run at least two full business weeks even if the sample target is hit sooner — and keep holiday anomalies out of the window.
One primary metric, declared up front
Track anything you like, but pre-declare the single metric that decides the test. Check enough secondary metrics "for significance" and one of them will oblige by pure chance.
Random split, frozen variants
Visitors must be randomly assigned and the split kept even — then leave the variants alone. Edit a live variant mid-test and the data from before and after the change can no longer be pooled.
Segments can flip the story
An overall winner can lose inside a key segment, and vice versa — Simpson's paradox is real on storefronts. Read the aggregate first, then confirm the win holds where the revenue actually lives.
Wins decay — verify them
Novelty flatters new designs for a few weeks. For big changes, re-check after the rollout with a holdback or a follow-up test to confirm the lift is structural, not curiosity.
Want the tests designed, built and read for you?
ONBRAND ships revenue-relevant changes as controlled experiments on 8, 9 and 10 figure storefronts — designed, gated, measured and cleaned up properly. Bring the hypothesis; we'll bring the rigor.