Mis on BigBird ja milliseid tähelepanu tüüpe see ühendab?

BigBird is a transformer architecture designed to process very long sequences, which traditional transformers cannot efficiently handle due to the quadratic complexity of attention.

BigBird combines several types of attention:

• Sparse attention: instead of full attention to all tokens, attention is limited to a subset of tokens.
• Global attention: some special tokens (e.g., CLS token) have access to all other tokens, allowing to preserve global information.
• Random attention: for each token, a random set of other tokens is selected for attention, helping to model diverse dependencies.
• Window attention: each token attends to neighboring tokens within a fixed window, providing local context.

The combination of these types of attention allows BigBird to scale efficiently to very long sequences, maintaining model quality and reducing computational costs from quadratic to linear or close to it.