What is logit bias and where is it used?

Logit bias is a method of modifying the probabilities of choosing words or tokens in language models (LLMs) by adding a bias to the logits — the values that are transformed into probabilities by the softmax function.

Applying logit bias allows controlling the likelihood of certain words or phrases appearing in the model's responses. For example, you can increase the probability of selecting desired terms or, conversely, decrease the likelihood of unwanted words.

This is used for:

• Controlling text generation (e.g., avoiding toxic words)
• Customizing the model's behavior for specific tasks
• Improving the quality and relevance of responses

Example: if the logit for the word "Java" is increased by a positive value, the model is more likely to choose this word in its response.