What is stemming and lemmatization in IR?

Stemming and lemmatization are methods of normalizing words in information retrieval (IR), which help to bring different forms of a word to a single base form to improve search and text analysis.

• Stemming is the process of cutting off suffixes and endings of words to get the root of the word. It often uses simple rules and can lead to somewhat incorrect or truncated forms. For example, the words "running", "runs", "runner" can be reduced to "run" or even "runn".

• Lemmatization is a more complex process that considers the context and grammatical form of the word to bring it to its dictionary form (lemma). For example, "better" will be reduced to "good", and "running" to "run".

In IR, these methods help to unify different forms of words, improving search relevance.