How do BatchNorm and LayerNorm work? Advantages and disadvantages of each and where they are applied.

BatchNorm (Batch Normalization) and LayerNorm (Layer Normalization) are methods of activation normalization in neural networks that help accelerate training and stabilize gradients.

BatchNorm:

• Normalizes the inputs of a layer across the batch: calculates the mean and variance for each feature over all examples in the batch.
• Formula: ( \hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} ), where (\mu_B) and (\sigma_B^2) are the mean and variance over the batch.
• After normalization, scaling and shifting are applied with learnable parameters (\gamma) and (\beta).

Advantages of BatchNorm:

• Speeds up convergence.
• Allows the use of higher learning rates.
• Often improves generalization.

Disadvantages of BatchNorm:

• Depends on batch size; performs poorly with small batches.
• Not suitable for recurrent networks and online learning.

LayerNorm:

• Normalizes across features within a single example, i.e., calculates the mean and variance over all neurons in the layer for each individual example.
• Independent of batch size.

Advantages of LayerNorm:

• Works well with recurrent networks (RNN, LSTM).
• Independent of batch size.

Disadvantages of LayerNorm:

• May be less effective in convolutional networks compared to BatchNorm.

Application:

• BatchNorm is widely used in convolutional neural networks (CNN).
• LayerNorm is often used in transformers and recurrent networks.

---

Example of using BatchNorm in PyTorch:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(100, 50),
    nn.BatchNorm1d(50),
    nn.ReLU()
)

Example of LayerNorm:

model = nn.Sequential(
    nn.Linear(100, 50),
    nn.LayerNorm(50),
    nn.ReLU()
)