Tell us about the architecture of ViT (Vision Transformer): how does an image turn into a sequence of tokens?

The Vision Transformer (ViT) architecture adapts the idea of transformers from NLP for image processing. The main idea is to represent an image as a sequence of tokens, similar to words in text.

The process of converting an image into a sequence of tokens:

1. Splitting the image into patches: the original image is divided into small square sections (e.g., 16x16 pixels).

2. Flattening patches: each patch is flattened into a vector (for example, if the patch is 16x16 and has 3 RGB channels, the vector size is 16163=768).

3. Projection into embeddings: these vectors pass through a linear layer (fully connected), which transforms them into fixed-dimensional vectors — tokens.

4. Adding positional information: since the transformer does not consider the order of tokens, positional embeddings are added to the embeddings to preserve information about the patches' positions in the image.

5. Feeding the sequence of tokens into the transformer: the resulting sequence of tokens is processed by a standard transformer (attention mechanism, normalization layers, etc.) to extract features and solve the task (e.g., classification).

Thus, ViT converts an image into a sequence of vectors that can be efficiently processed using transformer architecture, enabling the use of attention benefits for visual data analysis.