What is positional encoding in a transformer?

Positional encoding in transformers is a way to add information about the order of elements in a sequence, since transformers themselves do not have an inherent mechanism to account for token positions.

A transformer processes input as a set of tokens without considering their order, so a vector of positional encoding is added to the token embeddings, which depends on the position of the token in the sequence.

The most commonly used is sinusoidal positional encoding, where values are computed using sines and cosines of different frequencies for each position:

• For position pos and dimension i:
  • PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
  • PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

This allows the model to distinguish positions and generalize to sequences of different lengths.

Thus, positional encoding provides the transformer with information about the order of tokens, which is necessary for understanding the structure of the sequence.