What is familywise error?

Familywise error rate (FWER) is the probability of making at least one Type I error (false positive) when conducting multiple statistical tests simultaneously. In the context of A/B testing, if we test several hypotheses or metrics, the overall probability of falsely rejecting at least one true null hypothesis increases.

For example, if we perform 20 independent tests at a significance level of 0.05, the probability of getting at least one false positive is significantly higher than 5%. To control FWER, adjustments such as the Bonferroni method are used, which reduce the significance level for each individual test to keep the overall error probability at the desired level.