What is a Type I error (alpha) and a Type II error (beta)?

Type I error (alpha) is a false positive, when we reject the null hypothesis even though it is true. In simpler terms, we think there is an effect when there actually isn't.

Type II error (beta) is a false negative, when we fail to reject the null hypothesis even though it is false. That is, we do not notice a real effect.

For example, when testing a new feature on a website:

• Type I error — we think the new feature improved conversion, but in reality, there is no change.
• Type II error — we do not notice that the feature actually improved conversion.