Mida näitab tähelepanu vektor transformaators?

Attention vector in transformers shows how strongly the model "pays attention" to each word (or token) in the input sequence when processing a specific output token. Simply put, these are weights that determine the importance of each input element for the current step.

For example, when translating a sentence in a transformer, an attention vector is calculated for each output word, indicating which words in the input sentence are most relevant for generating that word. This allows the model to consider the context and dependencies between words, even if they are far apart.

The attention vector is a set of numbers (weights), usually normalized with softmax, where larger values indicate greater importance of the corresponding input token for the current step.

Example: if the model is translating the word "bank" in the context of "river bank", the attention vector may highlight the words "river" and "bank" in the input to correctly understand the meaning.