What are the problems with the Sigmoid activation function? Where is it used and how is it interpreted?

The Sigmoid activation function has the form:

[ \sigma(x) = \frac{1}{1 + e^{-x}} ]

It transforms any input value into the range (0, 1), which is convenient for interpretation as a probability.

Problems of Sigmoid:

• Vanishing gradients: For large magnitude input values, the derivative of the function is close to zero, which slows down the training of deep networks.
• Non-centered output: The output is always positive, which can slow down the convergence of gradient descent.
• Saturation: For large positive or negative values, the function "saturates," and gradients become very small.

Application:

• Often used in the output layer for binary classification tasks, where the output is interpreted as the probability of belonging to a class.

Interpretation:

The function value can be perceived as the probability of an object belonging to the positive class. For example, an output of 0.9 indicates a 90% probability.

Example of usage in Python:

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

output = sigmoid(2.0)  # approximately 0.88

In modern deep networks, ReLU and its variants are often preferred due to the problems associated with Sigmoid.