Võrrelge pre-norm ja post-norm transformereid. Milline on stabiilsem suure sügavusega õppimisel?

Pre-norm and post-norm are two options for placing the normalization layer (LayerNorm) in the transformer architecture.

• Post-norm (original version from Vaswani et al., 2017): normalization is applied after the attention or FFN block and after the residual connection is added.

• Pre-norm: normalization is applied before the attention or FFN block, and the residual connection is added after.

Differences and impact on training:

• Pre-norm transformers are generally trained more stably at greater depths (more layers). This is because normalization before the block helps better control gradients and prevents their vanishing or exploding.

• In post-norm architecture, increasing the number of layers often leads to gradient problems, making it difficult to train deep models.

Conclusion: for deep transformers (e.g., more than 12 layers), it is preferable to use pre-norm, as it provides more stable and efficient training.