Why does a network with BatchNorm converge faster?

Batch Normalization (BatchNorm) accelerates the convergence of neural networks by normalizing the inputs of each layer across mini-batches. This helps stabilize the distribution of activations and reduces the problem of internal covariate shift.

The main reasons why networks with BatchNorm converge faster:

• Stabilization of activation distribution: Normalizing layer inputs reduces changes in data distribution that pass to the next layer, facilitating training.
• Improved gradients: Normalization helps avoid vanishing or exploding gradients, making optimization more stable.
• Allows for larger learning rates: Thanks to stabilization, more aggressive optimization parameters can be used.
• Some regularization: BatchNorm has a regularizing effect, reducing overfitting.

In the end, BatchNorm makes the training process more stable and faster, leading to quicker network convergence.