What is speculative decoding and how does it speed up generation?

Speculative decoding is a method of speeding up text generation in large language models (LLMs), where the model simultaneously predicts several continuation options and then selects the most likely one or combines their results.

The idea is to run multiple "speculative" decoders (usually simpler and faster models) in parallel, which generate candidate continuations. The main, more accurate model then verifies these options and confirms or rejects them, allowing to avoid full sequential computation of each token.

Advantages:

• Reduced generation time through parallel processing.
• Lower computational costs, as the main model does not compute all options fully.

Thus, speculative decoding optimizes the generation process, speeds up output, and increases resource efficiency without significant loss of quality.