Why does regularization help fight overfitting?

Regularization helps combat overfitting by adding an extra penalty for model complexity to the loss function. This encourages the model to avoid overly complex solutions that fit the training data well but generalize poorly to new data.

Main reasons for the effectiveness of regularization:

• Reducing overfitting: Regularization constrains the magnitude of model parameters (e.g., weights in neural networks), preventing excessive fitting to noise in the data.
• Simplifying the model: By penalizing large coefficients, regularization promotes simpler and more robust models.

Example: L2 regularization (Ridge) adds the sum of squares of weights to the loss function:

[ Loss = Loss_{original} + \lambda \sum w_i^2 ]

where ( \lambda ) is the regularization coefficient controlling the penalty degree.

Thus, regularization helps the model to be more generalizable and resistant to new data.