What problems arise when initializing weights with zeros? Which initialization methods (Xavier, He) address these problems?

Initializing neural network weights with zeros leads to a symmetry problem: all neurons in a layer start learning the same, receive identical gradients, and update identically. As a result, the network cannot learn effectively because feature diversity does not develop.

Initialization methods such as Xavier (Glorot) and He address this issue by setting weights to random values with a specific distribution and scale, depending on the number of inputs and outputs of the neuron. This helps maintain the variance of signals and gradients at each layer, preventing vanishing or exploding gradients.

• Xavier Initialization is suitable for tanh and sigmoid activations. Weights are chosen from a distribution with a variance inversely proportional to the sum of incoming and outgoing connections.

• He Initialization is optimized for ReLU activations, where weights are scaled considering only the incoming connections, which helps better preserve gradients.

Example of weight initialization in PyTorch with He:

import torch
import torch.nn as nn

layer = nn.Linear(in_features=256, out_features=128)
nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')