What is the difference between self-attention and cross-attention, and what is each used for?

Self-attention is a mechanism where elements of an input sequence interact with each other to compute importance weights. Each element 'attends' to all other elements in the same sequence to understand which ones are more relevant for the current step. This is a key component in transformers for sequence processing, such as in language models.

Cross-attention is used when there are two different sequences: one acts as a query, and the other as keys and values. Cross-attention allows the model to focus on relevant parts of the second sequence when processing the first. For example, in translation or text generation tasks, cross-attention links the decoder output with the encoder output.

In summary:

• Self-attention helps the model consider context within a single sequence.
• Cross-attention links two different sequences, allowing one to incorporate information from the other.

Example: in a translation transformer, self-attention is used within the encoder and decoder, while cross-attention is used between the encoder and decoder.