Τι είναι ένα στρώμα FFN σε έναν μετασχηματιστή και γιατί είναι απαραίτητη η μη γραμμικότητα μεταξύ των μπλοκ προσοχής;

The FFN (Feed-Forward Network) layer in a transformer is a two-layer fully connected neural network, applied separately to each element of the sequence after the attention layer. Its task is to add additional nonlinear processing and increase the expressive capacity of the model.

Non-linearity between attention blocks is necessary for the model to approximate complex functions and dependencies. Attention is responsible for the weighted sum of information across different positions, and the FFN with nonlinear activation (such as ReLU or GELU) helps the transformer model more complex patterns than just linear combinations.

Example structure of an FFN layer:

import torch
import torch.nn as nn

class FFN(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.activation = nn.GELU()
        self.linear2 = nn.Linear(d_ff, d_model)

    def forward(self, x):
        return self.linear2(self.activation(self.linear1(x)))

Here, d_model is the dimension of input and output vectors, d_ff is the dimension of the internal layer, usually larger than d_model.

Thus, the nonlinear FFN layer provides a deep nonlinear transformation of features, improving training and model quality.