How does tokenization work? Why are subword tokens used instead of whole words?

Tokenization is the process of breaking down text into smaller parts — tokens, which can be words, subwords, or characters.

Subword tokens are used instead of whole words because:

• They allow efficient handling of unknown or rare words by breaking them into familiar parts.
• They reduce the size of the model's vocabulary, which decreases memory requirements and speeds up training.
• They improve the model's generalization, as it learns from parts of words rather than only whole words.

For example, the word "непрерывность" can be broken into tokens "не", "прерыв", "ность". If the model encounters a new word "непрерывный", it can recognize familiar parts and understand the meaning.

Algorithms like BPE (Byte Pair Encoding) or WordPiece build a subword token vocabulary based on the frequency of character sequence occurrences.