Τι είναι η μάσκα padding κλειδιού έναντι της μάσκας προσοχής στο PyTorch;

In PyTorch, when working with transformers and attention, masks are often used to correctly process sequences of different lengths and to prevent attention to unwanted positions.

• Key Padding Mask: is a mask that indicates which positions in the sequence are padding (fillers), so the model does not consider them when calculating attention. It is usually a boolean tensor where True indicates a padding position. It is applied to the keys and values in the attention mechanism to ignore padding tokens.

• Attention Mask: a more general mask that controls which positions can influence the current position during attention calculation. For example, in automatic text generation tasks, a mask is used to prevent looking ahead (future tokens) to avoid information leakage. This mask can be a triangular matrix where certain connections are forbidden.

Example:

import torch
from torch.nn import MultiheadAttention

# Suppose we have a sequence with padding
key_padding_mask = torch.tensor([[False, False, True, True]])  # True for padding

# Attention mask to prevent attention to future tokens
attn_mask = torch.triu(torch.ones(4, 4), diagonal=1).bool()  # Upper triangular mask

mha = MultiheadAttention(embed_dim=8, num_heads=2)
query = torch.rand(4, 1, 8)  # (sequence length, batch, embed_dim)
key = value = query

output, attn_weights = mha(query, key, value, attn_mask=attn_mask, key_padding_mask=key_padding_mask)

This way, the key padding mask is used to ignore padding tokens, and the attention mask to control the attention structure (e.g., preventing attention to future positions).