How to calculate sample size for A/B testing?

To calculate the sample size in an A/B test, several parameters need to be considered:

• Significance level (α) — the probability of incorrectly rejecting the null hypothesis (usually 0.05).
• Test power (1 - β) — the probability of detecting a real effect (usually 0.8 or 0.9).
• Expected effect (difference between groups) — the minimal change that is important to detect.
• Standard deviation or baseline metric level — to assess variability.

The formula for a two-proportion test is approximately:

[ n = \frac{2 (Z_{1-\alpha/2} + Z_{1-\beta})^2 p (1-p)}{d^2} ]

where:

• (Z_{1-\alpha/2}) and (Z_{1-\beta}) — quantiles of the normal distribution,
• (p) — average conversion rate,
• (d) — minimal difference between groups.

For convenience, online calculators or statistical libraries are often used.

An example in Python using the statsmodels library:

from statsmodels.stats.power import NormalIndPower

# Parameters
alpha = 0.05
power = 0.8
effect_size = 0.1  # for example, a 10% difference

analysis = NormalIndPower()
sample_size = analysis.solve_power(effect_size=effect_size, power=power, alpha=alpha, ratio=1)
print(f"Sample size per group: {int(sample_size)}")