How was BERT trained? What are MLM and NSP?

BERT (Bidirectional Encoder Representations from Transformers) was trained using two main tasks:

1. MLM (Masked Language Modeling) — masking words in a sentence and predicting these words based on the context. During training, a random portion of tokens (about 15%) is masked, and the model must predict which words were hidden using information from both the left and right (bidirectional context).

2. NSP (Next Sentence Prediction) — a task of predicting whether the second sentence in a pair is actually the next one following the first in the original text. This helps the model understand the relationship between sentences.

Thus, MLM teaches the model to understand the context within a sentence, and NSP — the relationships between sentences, which improves the quality of representations for natural language processing tasks.