What problems can arise when using the Sigmoid activation function together with BatchNorm?

When using the Sigmoid activation function together with Batch Normalization, the following issues may arise:

• Gradient compression: Sigmoid compresses input values into the range (0,1), which can lead to vanishing gradients, especially if inputs are in saturated regions of the function (close to 0 or 1).

• Distribution disruption: BatchNorm normalizes the inputs to a layer, but after applying Sigmoid, the distribution becomes heavily compressed and asymmetric, which can impair learning.

• Loss of BatchNorm advantages: BatchNorm works better with activations that have zero mean and symmetric range (e.g., ReLU or tanh). Sigmoid shifts outputs into the positive range, reducing normalization effectiveness.

For these reasons, it is often recommended to use ReLU or other activation functions with BatchNorm, and apply Sigmoid at the output layer where necessary (e.g., for binary classification).