What is the peeking problem in A/B testing?

Peeking problem in A/B tests is a situation where the experimenter repeatedly checks intermediate test results and makes a decision to stop or modify the test based on these preliminary data.

The problem is that frequent checks increase the likelihood of false positive results (Type I errors), meaning one might mistakenly conclude that one version is better when it is actually due to chance.

To avoid the peeking problem, methods such as significance level adjustments (e.g., corrections for multiple comparisons) are used, or fixed data analysis points are planned in advance.