What is Dirichlet smoothing in IR language models?

Dirichlet smoothing is a probability smoothing method in language models for information retrieval (IR), based on a Bayesian approach using a Dirichlet prior distribution.

In IR tasks, a language model estimates the probability of a term appearing in a document. The problem arises when the term is absent in the document, making the probability zero, which negatively impacts ranking.

Dirichlet smoothing adjusts the probability estimates by considering the term frequencies in the document collection (corpus) and a smoothing parameter (usually denoted as μ). The formula:

[ P_{smoothed}(w|d) = \frac{c(w,d) + \mu P(w|C)}{|d| + \mu} ]

where:

• ( c(w,d) ) — the count of term ( w ) in document ( d ),
• ( |d| ) — the length of the document,
• ( P(w|C) ) — the probability of the term in the collection,
• ( \mu ) — the smoothing parameter controlling the balance between the document and the collection.

Thus, Dirichlet smoothing helps avoid zero probabilities and improves the quality of document ranking in IR systems.