What is FlashAttention and what memory gains does it provide?

FlashAttention is an optimized algorithm for computing the attention mechanism in transformers, which significantly reduces memory consumption and accelerates computations compared to the classical approach.

The main idea of FlashAttention is to compute attention without explicitly storing large intermediate matrices (such as the attention weight matrix QK^T), using block processing and efficient use of GPU cache memory. This allows reducing peak memory usage by approximately 3-4 times, which is especially important when training large language models (LLMs) with long sequences.

Typically, if classical attention requires O(N^2) memory for a sequence of length N, FlashAttention reduces this load through on-the-fly computations and optimizations, enabling the deployment of models with larger contexts on the same GPUs.

Thus, FlashAttention is a way to make attention more memory- and speed-efficient, which is critical for large-scale models.