What is context length and why is it difficult to extend?

Context length — this is the maximum number of tokens (words or parts of words) that a large language model (LLM) can consider at once when generating or analyzing text.

The limitation on context length is related to transformer architecture, where the computational complexity of attention grows quadratically with input length. This leads to:

• High memory and time costs as context length increases.
• Hardware resource constraints.

Expanding context length is difficult because:

• It requires significantly more computational resources.
• It necessitates changing the model architecture or using special techniques (e.g., context compression, sliding window, recurrent mechanisms).

Therefore, context length is a trade-off between the ability to consider more information and performance limitations.