What is BatchNorm?

BatchNorm (Batch Normalization) is a method of normalizing the input data of a neural network layer across a mini-batch during training. It helps accelerate and stabilize the training process of deep neural networks by reducing the problem of internal covariate shift.

The main idea is to compute the mean and standard deviation for each feature in the mini-batch, normalize the data, and then apply scaling and shifting with learnable parameters:

$$\hat{x} = \frac{x - \mu_{batch}}{\sqrt{\sigma_{batch}^2 + \epsilon}}$$ $$y = \gamma \hat{x} + \beta$$

where $\mu_{batch}$ and $\sigma_{batch}^2$ are the mean and variance over the batch, and $\gamma$ and $\beta$ are the scale and shift parameters, learned together with the network.

BatchNorm helps to:

• Accelerate the convergence of training
• Allow the use of higher learning rates
• Reduce the need for other forms of regularization

Example of usage in PyTorch:

import torch
import torch.nn as nn

batch_norm = nn.BatchNorm1d(num_features=100)  # for 1D data with 100 features
input = torch.randn(32, 100)  # batch of 32 samples
output = batch_norm(input)

Thus, BatchNorm is an important tool for improving the quality and speed of neural network training.