What is TF-IDF?

TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical measure used to evaluate the importance of a word in a document relative to the entire corpus of documents.

• TF (Term Frequency) — the frequency of a term in a specific document, usually calculated as the number of occurrences of the word normalized by the length of the document.
• IDF (Inverse Document Frequency) — the inverse frequency of the document, reflecting how rare the word is across the entire corpus. It is calculated as the logarithm of the ratio of the total number of documents to the number of documents containing the word.

The final TF-IDF value for a word in a document is the product of TF and IDF. A high value indicates that the word appears frequently in this document but rarely in others, making it a good feature for highlighting the document's topic.

Example in Python using sklearn:

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["machine learning and artificial intelligence",
        "artificial intelligence and deep learning"]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(docs)

print(vectorizer.get_feature_names_out())
print(tfidf_matrix.toarray())