How does top-k differ from top-p (nucleus) sampling?

Top-k and top-p (nucleus) are sampling methods from probability distributions, often used for text generation in language models.

• Top-k sampling selects from the k most probable tokens. Only the k tokens with the highest probability are taken from the entire vocabulary, and sampling occurs only among them. This limits the choice and prevents selecting rare tokens.

• Top-p (nucleus) sampling selects the minimal set of tokens whose cumulative probability exceeds a threshold p (e.g., 0.9). Thus, the number of tokens for sampling is dynamic and depends on the probability distribution. This allows for flexible consideration of more probable tokens, but not a fixed number.

Example:

• With top-k=5, always the 5 most probable tokens are chosen.
• With top-p=0.9, the minimal set of tokens whose combined probability is ≥ 0.9 is selected, which could be 3 or 10 tokens.

Thus, top-k fixes the number of candidates, while top-p sets a probability threshold for candidate selection.