What is multi-query attention and where is it used?

Multi-query attention is a modification of the attention mechanism in transformers, where multiple keys and values are combined into a single shared set, and each query uses a separate set of queries. This allows for a significant reduction in memory usage and computational costs, especially when working with long sequences, while maintaining the model's quality.

The main application of multi-query attention is in large language models and transformers where efficiency and scalability are important, such as in GPT-4 and other modern architectures. This approach helps to speed up inference and reduce resource requirements without a substantial loss of accuracy.

Example: instead of having separate keys and values for each attention head (multi-head attention), all heads share a common set of keys and values, while queries remain individual.