What is an OOV token and how does subword tokenization handle it?

OOV token (Out-Of-Vocabulary) is a special token used to denote words or symbols that are absent from the model's dictionary. Such words are not recognized directly, which can degrade the quality of text processing.

Subword tokenization addresses the OOV problem by breaking words into smaller parts — subwords or morphemes. Instead of considering the whole word, the model works with known subwords, allowing it to process new or rare words by combining familiar parts.

For example, the word "unhappiness" can be broken into subwords: "un", "happi", "ness". Even if "unhappiness" is not in the dictionary, the model understands it through familiar parts.

Thus, subword tokenization reduces the number of OOV tokens and improves the generalization ability of NLP models.