What is bias-variance trade-off?

Bias-variance trade-off is a fundamental concept in machine learning that describes the trade-off between two types of errors when training a model:

• Bias — error caused by a model being too simple to capture the underlying data patterns (underfitting).
• Variance — error related to a model being too sensitive to fluctuations in the training data, leading to poor generalization on new data (overfitting).

If the model is too simple, it will have high bias and low variance. If it is too complex, it will have low bias but high variance.

The goal is to find a balance where the model is complex enough to capture the patterns but not so complex that it overfits noise.

Example: when training a polynomial regression of low degree (linear regression), bias will be high and variance low; with a high-degree polynomial, the opposite occurs.