Why does L1 regularization zero out some weights, but L2 does not?

L1 regularization (Lasso) adds the sum of absolute weight values ( \lambda \sum |w_i| ) to the loss function, while L2 regularization (Ridge) adds the sum of squares of weights ( \lambda \sum w_i^2 ).

Mathematically:

• L1 regularization results in a non-differentiable point at zero (not differentiable at 0), which encourages some weights to become exactly zero, as the optimization "tends" towards points where weights are exactly zero.
• L2 regularization has a smooth quadratic form, with the derivative equal to (2\lambda w_i), leading to a gradual reduction of weights, but not their exact zeroing.

Geometrically, with L1 penalty, the feasible solution region forms a diamond shape with sharp corners, increasing the likelihood of solutions with zero weights. In the case of L2 penalty, the region is circular, which does not promote zero weights.

Example:

\text{L1 penalty: } \lambda |w| \rightarrow \text{derivative is undefined at } w=0
\text{L2 penalty: } \lambda w^2 \rightarrow \text{derivative } 2\lambda w

Therefore, L1 regularization is used for feature selection, while L2 is used for weight smoothing.

L1 regularization (Lasso) adds the sum of absolute values of weights (||w||₁) to the loss function, which leads to corner points in the optimization surface. This encourages the optimal solution to often have exactly zero values for some weights, effectively zeroing out some of them and ensuring model sparsity.

L2 regularization (Ridge) adds the sum of squares of weights (||w||₂²), creating a smooth parabolic surface without corners. As a result, weights decrease but rarely become exactly zero, as the optimization tends to small but non-zero values.

Example:

import numpy as np
from sklearn.linear_model import Lasso, Ridge

X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1, 2, 3])

lasso = Lasso(alpha=0.1)
lasso.fit(X, y)
print('L1 weights:', lasso.coef_)

ridge = Ridge(alpha=0.1)
ridge.fit(X, y)
print('L2 weights:', ridge.coef_)

Here, L1 can zero out some weights, while L2 only reduces them.