What is CUPED and how does it reduce variance?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a method for reducing variance in A/B tests and experiments, which uses data collected before the experiment begins to control for variations.

The idea is that users have pre-experiment metrics (e.g., behavior before the test starts) that correlate with the target metric. Using this data, results can be adjusted to reduce noise and increase the statistical power of the test.

How it works:

1. A pre-experiment metric (covariate) is collected, for example, the average number of purchases in a week before the test.
2. A corrected metric for each user is calculated by subtracting the influence of the pre-experiment metric.
3. The corrected metric is analyzed, which reduces variance and allows detecting effects with fewer users.

Example formula:

[ Y_i^{adj} = Y_i - \theta (X_i - \bar{X}) ]

where:

• (Y_i) — observed metric in the experiment for user i
• (X_i) — pre-experiment metric
• (\bar{X}) — average pre-experiment metric
• (\theta) — regression coefficient between (Y) and (X)

Thus, CUPED helps improve the accuracy of effect estimation, reducing the influence of random fluctuations and enhancing the quality of conclusions.