How is statistical significance calculated in A/B testing?

Statistical significance in conversion rate A/B testing evaluates whether the observed performance difference between Control (A) and Treatment (B) is real or caused by random sampling variance. Using a two-tailed two-proportion Z-test, the standard error of the pooled proportion SE = sqrt(p_pool * (1 - p_pool) * (1/Na + 1/Nb)) is used to compute the Z-score Z = (pB - pA) / SE. The two-tailed p-value represents the probability of observing a difference this extreme under the null hypothesis (p-value = 2 * (1 - Φ(|Z|))).

Why is 95% confidence (alpha = 0.05) the standard benchmark?

A 95% confidence level establishes a 5% false positive rate (alpha = 0.05). This means there is only a 1-in-20 probability of declaring a winner when no true performance difference exists. High-risk product decisions often demand 99% confidence (alpha = 0.01), while early exploratory tests may use 90% confidence.

What is the difference between relative lift and absolute difference?

Absolute difference is the simple subtraction of conversion rates: pB - pA (e.g., 3.8% - 3.2% = 0.6% percentage points). Relative lift measures the percentage increase relative to the baseline control: (pB - pA) / pA * 100% (e.g., (0.038 - 0.032) / 0.032 = +18.75% lift).

How does Minimum Detectable Effect (MDE) impact required sample size?

The required sample size is inversely proportional to the square of the Minimum Detectable Effect (MDE). Detecting a small 2% lift requires roughly 25 times more visitors than detecting a 10% lift. Choosing an overly ambitious MDE leads to underpowered tests and false negatives.

Why shouldn't you stop an A/B test early as soon as it reaches p < 0.05?

Stopping a test early because p < 0.05 is known as the 'peeking problem' or continuous monitoring fallacy. It dramatically inflates the true false positive rate from 5% up to 30–40% due to repeated significance testing. Tests must run until reaching the pre-determined sample size and at least one full business cycle (typically 1 to 2 weeks).

A/B Testing Statistical Significance & Sample Size Engine (2026)

Q: Why shouldn't you stop an A/B test early as soon as it reaches p < 0.05?

Stopping a test early because p < 0.05 is known as the 'peeking problem' or continuous monitoring fallacy. It dramatically inflates the true false positive rate from 5% up to 30–40% due to repeated significance testing. Tests must run until reaching the pre-determined sample size and at least one full business cycle (typically 1 to 2 weeks).

Direct Statistical Significance & Z-Test Formula

A/B Testing Statistical Significance evaluates whether the conversion difference between Control (A) and Treatment (B) is mathematically real or random variance. Using a two-tailed pooled Z-test, pooled proportion is p_pool = (Conversions_A + Conversions_B) / (N_A + N_B). The test statistic is formulated as Z = (p_B - p_A) / sqrt(p_pool × (1 - p_pool) × (1/N_A + 1/N_B)), yielding an exact p-value via the Gauss error function: p = 2 × (1 - Φ(|Z|)).

A result is declared statistically significant when p-value < α (e.g., α = 0.05 for 95% confidence). Required sample size per variation follows Evan Miller's formula: N = 2 × (Z_{α/2} + Z_β)² × p̄(1 - p̄) / δ², where δ represents the Minimum Detectable Effect (MDE). Running tests until reaching this exact pre-calculated sample size eliminates the false-positive peeking fallacy.

1. Split-Test Experiment Data

Real-time two-tailed Z-test

Control Group (Variation A)

Visitors (Sample N_A)

Unique visitors in Control.

Conversions (C_A)

✓

Goal completions in Control.

Treatment Group (Variation B)

Visitors (Sample N_B)

Unique visitors in Treatment.

Conversions (C_B)

✓

Goal completions in Treatment.

Statistical Parameters

Confidence Level (%)

Threshold to reject H₀.

Statistical Power (1 - β)

Probability of detecting a true effect.

Target Minimum Detectable Effect / MDE (%)

±%

Minimum expected relative lift you aim to detect reliably.

2. Statistical Significance Output

Statistically Significant Winner (95% Confidence)

Observed Relative Lift

+19.81%

Variation B vs Control A

P-Value (Two-Tailed)

0.0064

Z-Score: 2.7283

Control (A) Conversion Rate 3.2%

Treatment (B) Conversion Rate 3.83%

Chance to Beat Control 99.7%

95% Confidence Interval (Relative) [+5.59%, +34.04%]

Sample Size Requirement (MDE: 15.0%)

22,632 / variant

55.6% of target sample reached 45,264 total visitors needed

Variant B outperformed Variant A with a +19.81% lift (p-value = 0.0064 < 0.05). You can safely roll out Variant B to 100% of traffic.

🔍 Optimize the Rest of Your Funnel:

→ See how conversion lift scales Cohort Gross Margin & ARPU → Simulate Net Profit with 2D Sensitivity Matrix (CPC vs CR)

The Mathematical Framework of Conversion A/B Testing

Online experimentation relies on rigorous hypothesis testing to protect commercial platforms from rolling out noisy, false-positive variations:

1. Pooled Standard Error

SE = √(p̂(1 - p̂)(1/N_A + 1/N_B))

Weights variance across both cohorts under the null hypothesis (H₀: p_A = p_B).

2. Two-Tailed Z-Score & P-Val

Z = (p_B - p_A) / SE

p-value = 2(1 - Φ(|Z|)). If p < α, we reject the null hypothesis.

3. Sample Size (Evan Miller)

N = 2(Z_{α/2} + Z_β)² p̄(1-p̄) / δ²

Calculates minimum sample size required before launching the experiment.

Minimum Required Sample Sizes by Baseline CR & MDE (95% Conf, 80% Power)

Estimated sample size per variation required to reliably detect relative improvements without premature stoppage:

Baseline CR	MDE: ±5% Lift	MDE: ±10% Lift	MDE: ±15% Lift	MDE: ±20% Lift	Feasibility on 10k Visits/Mo
1.0%	1,550,000 / variant	388,000 / variant	172,000 / variant	97,000 / variant	Requires High-Traffic Platform
3.0% (Average E-com)	508,000 / variant	127,000 / variant	56,500 / variant	31,800 / variant	Feasible in 2–4 Months
5.0% (High-Intent Lead Gen)	298,000 / variant	74,500 / variant	33,200 / variant	18,700 / variant	Feasible in 1–2 Months
10.0% (Trial / Micro-step)	141,000 / variant	35,300 / variant	15,700 / variant	8,850 / variant	Rapid Turnaround (<3 Weeks)

Frequently Asked Questions

What is a two-tailed test and why is it preferred in CRO?

A two-tailed test evaluates both possibilities: whether Treatment B is significantly better OR significantly worse than Control A. A one-tailed test only checks if B is better, ignoring the possibility that the new variant degrades revenue. Two-tailed testing is the gold standard because it guards against costly negative deployments.

What does the p-value really tell you?

The p-value is the probability of obtaining test results at least as extreme as the observed data, assuming that the null hypothesis (no true difference between A and B) is completely true. A p-value of 0.006 means there is only a 0.6% chance that random noise alone produced this conversion difference.

What is the "Peeking Problem" in split testing?

Repeatedly inspecting an active test and stopping the moment p < 0.05 is mathematically flawed. Random variance naturally oscillates across significance boundaries early in experiments. Peeking inflates your true false positive rate from 5% to over 30%, causing teams to deploy illusory "winners" that regress to the mean post-launch.

How does Minimum Detectable Effect (MDE) affect business decision-making?

MDE represents the smallest lift that matters to your business. If a 1% lift translates to $2,000,000 in annual revenue, spending 6 months to detect it is justified. For smaller websites, targeting an MDE of 15% to 25% keeps required sample sizes manageable and accelerates testing velocity.

A/B Testing Statistical Significance Engine