What is weight tying between embedding and output projection?

Weight tying is a technique in neural networks, especially in models for natural language processing (e.g., transformers), where the weights of the embedding layer (word embeddings) and the output projection layer (projection onto the vocabulary during text generation) are shared and use the same parameters.

The idea is that the weight matrix that transforms input word indices into vectors (embedding) and the matrix that transforms the model's internal representation back into word probabilities (output projection) are essentially related and can be the same. This reduces the number of model parameters and improves generalization.

Example: if you have an embedding matrix of size (vocab_size × embedding_dim), then the output projection layer can use the transpose of this same matrix to compute logits before softmax.

This helps to:

• Reduce the number of parameters
• Improve convergence and model quality

In classical transformers like GPT, weight tying is often applied between the input embedding and the output linear layer.