Why do transformers have multiple attention heads? What do they learn in practice?

In transformers, multi-head attention is used to allow the model to focus on different parts of the input sequence simultaneously and extract diverse dependencies.

Each attention head is trained to highlight different aspects of information, for example:

• One head may focus on syntactic relationships between words.
• Another on semantic relationships.
• A third on positional or contextual features.

This enables the model to better understand complex structures and contexts in data, improving prediction quality.

For example, in a translation task, one head might track which word in the source sentence the current word in the translation relates to, while another considers the overall context of the phrase.

Thus, multiple attention heads provide a richer and more multifaceted representation of information compared to a single head.