Τι είναι το GELU και γιατί είναι καλύτερο από το ReLU στους μετασχηματιστές;

GELU (Gaussian Error Linear Unit) is an activation function frequently used in transformers and other modern neural networks. It differs from the classic ReLU in that it is smooth and considers the probabilistic nature of activation.

The approximate formula for GELU looks like this:

[ \text{GELU}(x) = x \cdot \Phi(x) ]

where ( \Phi(x) ) is the cumulative distribution function of the standard normal distribution.

Advantages of GELU over ReLU:

• Smoothness: GELU is continuous and differentiable, which improves learning.
• Stochasticity: GELU takes into account the probability of neuron activation, helping the model better handle noise and uncertainty.
• Better data representation: in NLP tasks and transformers, GELU promotes more efficient learning and improves model quality.

Example of usage in PyTorch:

import torch
import torch.nn.functional as F

def gelu(x):
    return 0.5 * x * (1 + torch.tanh(torch.sqrt(torch.tensor(2 / 3.1415)) * (x + 0.044715 * x ** 3)))

x = torch.tensor([-1.0, 0.0, 1.0])
print(gelu(x))

Thus, GELU provides a smoother and more informative activation compared to ReLU, which is important for complex models such as transformers.