Explain the scaled dot-product attention formula. Why divide by sqrt(d_k)?

Scaled dot-product attention is a mechanism that computes weights for sequence elements based on the scalar products of queries (Q) and keys (K). The formula looks like this:

[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V ]

Where:

• Q — query matrix,
• K — key matrix,
• V — value matrix,
• (d_k) — dimension of key vectors.

The division by (\sqrt{d_k}) is necessary for scaling the result of the scalar product. Without this step, the values can become too large when dimensions are high, leading to very large softmax arguments. This causes the "vanishing gradients" problem — softmax becomes very "sharp", almost one element gets weight 1, and others almost 0, which worsens training. Scaling stabilizes gradients and improves model convergence.