How to calculate the perplexity of a language model? What are its properties?

Perplexity is a metric of language model quality that measures how well the model predicts a sequence of words. Formally, perplexity is defined as the exponential of the average negative log-likelihood of the predicted words:

[ \text{Perplexity} = \exp\left(-\frac{1}{N} \sum_{i=1}^N \log P(w_i | w_{1:i-1})\right) ]

where (N) is the length of the sequence, and (P(w_i | w_{1:i-1})) is the probability of the word (w_i) given the previous words.

Properties of perplexity:

• The lower the perplexity, the better the model predicts the text.
• Perplexity can be interpreted as the average number of options the model considers when choosing the next word.
• Perplexity is always positive and usually greater than or equal to 1.
• It is sensitive to the size of the vocabulary and the length of the test set.

Example of calculating perplexity for a simple model in Python:

import math

# Suppose we have probabilities of words in a sequence
probs = [0.1, 0.5, 0.2, 0.2]

log_prob_sum = sum(math.log(p) for p in probs)
N = len(probs)
perplexity = math.exp(-log_prob_sum / N)
print(f"Perplexity: {perplexity}")