A/B Test Calculator

Statistical significance tells you the probability that the difference between control and variant is real, not random. 95% confidence means only a 5% chance (p < 0.05) the difference is noise. 99% confidence is used for high-stakes decisions. Running tests for too little time or traffic produces misleadingly significant results.

At least one full business cycle (1–2 weeks) to account for day-of-week effects. Never stop when you first reach 95% significance — the "peeking problem" inflates false positive rates. Calculate required sample size before starting, then run until you hit it regardless of interim results.

p-value is the probability of seeing a difference this large if no real difference exists. p=0.05 means 5% chance of a false positive (95% confidence). Lower p-value = stronger evidence. A significant p-value does NOT indicate practical importance — a 0.1% lift may be statistically significant but not worth acting on.

Depends on: (1) baseline conversion rate, (2) minimum detectable effect (MDE — the smallest lift worth acting on), (3) desired confidence and power. At 3% baseline and 20% MDE, you need ~5,000 visitors per variant. At 10% MDE: ~20,000 per variant. Underpowered tests produce unreliable results.