What does the attention vector in transformers show?

The attention vector in transformers indicates how much the model 'attends' to each word (or token) in the input sequence when processing a specific output token. In simpler terms, these are weights that determine the importance of each input element for the current processing step.

For example, when translating a sentence in a transformer, an attention vector is computed for each output word, indicating which words in the input sentence are most relevant for generating that word. This allows the model to consider context and dependencies between words, even if they are far apart.

The attention vector is a set of numbers (weights), usually normalized with softmax, where larger values indicate greater importance of the corresponding input token for the current step.

Example: if the model translates the word "bank" in the context of "river bank," the attention vector may highlight the words "river" and "bank" in the input to correctly understand the meaning.