How to obtain embeddings from text that will be used as input to a transformer?

To obtain embeddings from text for a transformer, the following steps are usually performed:

1. Tokenization: splitting the text into tokens (words, subwords, or characters), for example, using Byte-Pair Encoding (BPE) or WordPiece.
2. Converting tokens to indices: each token is mapped to a numerical index from the model's vocabulary.
3. Embedding tokens: the indices are fed into an embedding layer, which transforms them into fixed-dimensional vectors.

Example in Python using the Hugging Face Transformers library:

from transformers import AutoTokenizer, AutoModel
import torch

# Load pre-trained tokenizer and model
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModel.from_pretrained('bert-base-uncased')

text = "Example text for embeddings"

# Tokenization and conversion to tensors
inputs = tokenizer(text, return_tensors='pt')

# Getting embeddings (output of the last hidden layer)
with torch.no_grad():
    outputs = model(**inputs)

embeddings = outputs.last_hidden_state  # shape [batch_size, seq_len, hidden_size]

These embeddings can be used as input for further transformer layers or other models.