Evan Miller Calculator for A/B Tests

Calculating Sample Size for A/B Tests: Evan Miller's Calculator

Evan Miller's calculator allows you to accurately determine the sample size needed for A/B tests, minimizing the risk of errors and optimizing resources. The tool takes into account the baseline conversion rate, expected effect size, statistical power, and significance level, outputting the required number of observations for groups A and B.

Proper calculation prevents underestimating the sample size, which leads to high variability, or wasting resources on excessive data volume. For example: with a baseline conversion rate of 56% and a goal of increasing it by 3%, approximately 13,719 users per group are required under standard parameters (80% power, α=5%). A 20% buffer is recommended, bringing the total to 16,463 per group.

Calculation Parameters in the Calculator

The tool requires sequential input of key metrics:

• Baseline conversion rate — the current conversion rate. Calculated as (successful events / total events). Example: 280,000 views out of 500,000 emails sent = 56%.
• Minimum Detectable Effect (MDE) — the minimum effect size you want to detect. Specified in absolute (units) or relative (%) values. For conversion rates, relative values are preferred.
• Statistical power (1-β) — the probability of detecting a real effect (standard is 80%). A higher value requires a larger sample size.
• Significance level (α) — the probability of a Type I error (false positive, standard is 5%).

A graph in the interface shows the range of effect sizes that would confirm the hypothesis (gray interval from baseline + MDE).

How Parameters Affect Sample Size

Adjusting the settings directly impacts the required data volume:

• Increasing power from 80% to 90% raises the sample size from 13,719 to 18,372 per group.
• A lower α (e.g., 1%) requires an even larger volume.
• Relative MDE is easier to interpret for percentage-based metrics, while absolute MDE is better for fixed changes.

| Parameter | Standard Value | Effect on Sample Size |

|-----------|----------------|-----------------------|

| Power | 80% | Baseline |

| α | 5% | Baseline |

| Power 90% | 90% | +34% |

| α 1% | 1% | +~100% |

After calculation, determine test duration: sample size / average daily traffic. Budget: sample size × cost per acquisition.

Practical Recommendations for Use

• Add a 20% buffer to account for user churn or traffic fluctuations.
• Split traffic 50/50 between groups using A/B testing platforms (Optimizely, Google Optimize).
• For binary metrics (conversion), use relative MDE.
• Validate assumptions: consider seasonality and audience segmentation.

Example Python code for verification (using statsmodels):

import statsmodels.stats.api as sms

effect_size = sms.proportion_effectsize(0.56, 0.56 * 1.03)
analysis = sms.NormalIndPower()
sample_size = analysis.solve_power(effect_size, power=0.8, alpha=0.05, ratio=1)
print(f"Sample size per group: {sample_size:.0f}")

Output: ~13,719, matching the calculator.

Key Takeaways

• Accurate sample size calculation reduces the likelihood of Type I and Type II errors.
• Standard parameters (80% power, 5% α) balance speed and reliability.
• A 20% buffer compensates for real-world data loss.
• Integration with code allows for automated validation.
• Traffic considerations determine the actual test duration.

— Editorial Team