What is text normalization (NFC, NFKC) and why should it be done before tokenization?

Text normalization (NFC, NFKC) is the process of converting Unicode characters into a standard form so that different representations of the same character become identical.

• NFC (Normalization Form C) is a composite form where characters consisting of a base character and a diacritic mark are combined into a single character.
• NFKC is an extended composite form that additionally considers compatible characters, bringing them to a canonical equivalent.

Why normalize before tokenization:

1. Simplifies text processing by eliminating ambiguities.
2. Allows correct comparison and searching of words.
3. Ensures consistency of data for models and algorithms.

For example, the letter "é" can be written as a single character (U+00E9) or as 'e' + accent (U+0065 + U+0301). Without normalization, these variants will be perceived as different.