What are Medusa and multi-token prediction heads?

Medusa is an architecture or method in the field of large language models (LLM) aimed at optimizing the text generation process. It allows efficiently predicting multiple tokens (words or characters) at once, speeding up output and reducing computational costs.

Multi-token prediction heads are components of the model that predict several next tokens simultaneously, rather than one. In classical transformers, the prediction head usually outputs probabilities for the next token, but multi-token heads extend this idea to generate blocks of text in one step. This improves performance and reduces latency during generation.

For example, this can be represented as:

# Pseudocode for multi-token prediction
input_sequence = "Hello, how"
# Model with multi-token head predicts several tokens at once
predicted_tokens = model.predict_next_tokens(input_sequence, num_tokens=3)
print(predicted_tokens)  # e.g., ['are', '?', ' today']

Thus, Medusa and multi-token prediction heads are approaches to accelerate and optimize text generation in LLMs by predicting multiple tokens at once.