What is a token in the context of LLM and embedding?

In the context of large language models (LLMs) and embeddings, a token is the smallest unit of text that the model processes. Tokens can be words, parts of words, or even individual characters, depending on the tokenizer used.

For example, the word "programming" can be broken down into multiple tokens if the tokenizer uses subword units.

Tokens serve as input elements for the model: it takes a sequence of tokens and generates output data or embeddings based on them — numerical vectors that reflect the meaning of the text.

Example: for the text "Hello, world!" the tokenizer might produce tokens ["Hello", ",", "world", "!"] or break them into smaller parts.

Thus, a token is a basic building block for text processing in LLMs and when creating embeddings.