How does stemming differ from lemmatization? In which tasks should each be used?

Stemming and lemmatization are methods of normalizing words in natural language processing (NLP), but they differ in approach and accuracy.

Stemming is a rough truncation of words to their root form (stem) by removing suffixes and endings. It does not consider context and may cut words incorrectly, sometimes resulting in non-existent roots.

Lemmatization is a more precise process that reduces a word to its dictionary form (lemma), considering morphological analysis and part of speech. For example, "бегу" and "бегал" are lemmatized to "бежать".

When to use:

• Stemming is suitable for tasks where speed is important and some accuracy can be sacrificed, such as in search engines or large-scale text analysis.
• Lemmatization is preferred when accuracy and semantic correctness are crucial, such as in machine translation, sentiment analysis, or information extraction.

Example of stemming in Python using the NLTK library:

from nltk.stem import PorterStemmer
ps = PorterStemmer()
print(ps.stem('running'))  # output: run

Example of lemmatization:

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('running', pos='v'))  # output: run