Mis on CUPED ja kuidas see vähendab variatsiooni?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a method to reduce variance in A/B tests and experiments, which uses data collected before the experiment to control variations.

The idea is that users have pre-experiment metrics (e.g., behavior before the test starts) that correlate with the target metric. Using this data, results can be adjusted, noise reduced, and the statistical power of the test increased.

How it works:

1. A pre-experiment metric (covariate) is collected, for example, the average number of purchases per week before the test.
2. For each user, an adjusted metric is calculated by subtracting the influence of the pre-experiment metric.
3. The adjusted metric is analyzed, which reduces variance and allows detection of effects with fewer users.

Example formula:

[ Y_i^{adj} = Y_i - \theta (X_i - \bar{X}) ]

where:

• (Y_i) is the observed metric in the experiment for user i
• (X_i) is the pre-experiment metric
• (\bar{X}) is the mean of the pre-experiment metric
• (\theta) is the regression coefficient between (Y) and (X)

In this way, CUPED helps improve the accuracy of effect estimation, reducing the impact of random fluctuations and enhancing the quality of conclusions.