Masked Language Modeling, dynamic masking, and the lesson that sometimes the advance is training better what already existed.
In October 2018, four months after GPT-1, Google published BERT. It started from the same Transformer block and made one different decision:
Without the mask, each token sees the whole sentence — past and future. The representation of "banco" in "sentei no banco da praça" (Portuguese: "I sat on the bench in the square") is now built with "praça" available from the very first block.
The gain in comprehension is enormous and immediate. BERT broke records on eleven tasks at once. But there is a price, and it is fatal for one specific thing.
If each token sees all the others, the objective of predicting the next word becomes trivial. To predict word 5, just look at word 5. The model would learn to copy, and nothing else.
Note that this is exactly the label leakage the causal mask existed to prevent (chapter 04). By removing the mask, BERT lost the right to use GPT's objective. It needed another training objective.
BERT's idea is elegant: instead of hiding the future, hide randomly drawn words, scattered through the text, and ask the model to reconstruct them using everything left on both sides.
Original text:
The cat sat on the mat and slept peacefully
Model input (15% of tokens masked):
The cat <mask> on the mat and <mask> peacefully
^
Target: reconstruct 'sat' and 'slept'.
To get 'sat' right, the model uses 'cat' (on the left) AND 'on the mat'
(on the right). That is what a BIDIRECTIONAL task means.
This is called Masked Language Modeling (MLM). In plain terms: the fill-in-the-blank task, the sentence-completion exercise you did at school. The model is forced to understand the whole context to fill the gap.
Of the 15% of tokens selected for masking, BERT does not replace all of them with <mask>. It does:
| Fraction | What it does | Why |
|---|---|---|
| 80% | replaces with <mask> | the main task |
| 10% | replaces with a random token | forces the model to distrust every token, not just the masked ones. Without this, it would learn to blindly copy any unmasked token. |
| 10% | leaves the original token | prevents the model from learning "I only need to think when I see <mask>". It needs to keep a good representation of every token, always. |
Compare the efficiency. In GPT-2, a sequence of
That is why MLM models need many more passes over the data to reach the same learning — and it is one of the reasons the decoder family ended up dominating once scale became the decisive factor.
There is also a mismatch between training and use: the <mask> token appears in training and never in real applications. The 80/10/10 is precisely a patch to reduce that mismatch.
RoBERTa is not a new architecture. It is the same BERT, trained properly. The name is the acronym of Robustly Optimized BERT Approach, and the paper from the Facebook AI authors has an implicit subtitle that is almost a provocation: BERT was significantly undertrained.
| Change | BERT | RoBERTa |
|---|---|---|
| Masking | static — the masks are drawn once, in preprocessing, and the model always sees the same pattern for each sentence | dynamic — redrawn every epoch. The same sentence is seen with different gaps. |
| NSP task | predicted whether two sentences were consecutive | removed — the authors showed it hurt |
| Data | 16 GB | 160 GB (10×) |
| Batches | 256 sequences | 8,000 sequences |
| Tokenizer | WordPiece, 30,000 | GPT-2's byte-level BPE, 50,265 |
| Training | 1 M steps | 500 K steps with much larger batches |
In the original BERT, the sentence "O gato sentou no tapete" ("The cat sat on the mat") had its masks decided a single time. If the draw picked "sentou", the model would always see that same gap, epoch after epoch — and could simply memorize that specific pair.
In RoBERTa, every epoch redraws. In one epoch the gap is "sentou", in another it is "tapete", in another it is "gato". Each sentence becomes multiple different exercises, and the model is forced to build a representation that serves to reconstruct any part. It is cheaper than collecting new data, and it works.
BERT had a second objective: given a pair of sentences, predict whether the second followed the first in the original document (Next Sentence Prediction). The intuition was to teach relations between sentences, useful for tasks like textual inference.
The RoBERTa authors tested it and showed that NSP did not help — and in several tasks, hurt. The task was too easy: telling two sentences from the same document apart from two sentences from random documents is almost always a matter of topic, not coherence. The model learned to detect a change of subject, a shallow signal, and spent capacity on it.
RoBERTa invented nothing. It took an existing architecture, questioned every training decision, measured, and removed what did not work. It gained several points on benchmarks and became the foundation of a good part of the NLP industry.
In a field that tends to value new architectures, this paper is an important reminder: a considerable share of what gets attributed to "better architecture" is, in fact, better training — and it is almost never measured separately. It is worth being suspicious of this whenever you read a paper announcing a new architecture.
| GPT-2 | RoBERTa | |
|---|---|---|
| Causal mask | yes | no |
| Training objective | predict the next token | reconstruct masked tokens |
| Signal per sequence | 100% of positions | 15% of positions |
| Special tokens | <|endoftext|> | <s> </s> <pad> <unk> <mask> |
| Wraps the sentence? | no | <s> … </s> |
| Vocabulary | 50,257 | 50,265 |
| Context | 1024 | 512 |
| Generates text? | yes | no |
| KV cache possible? | yes | no |
RoBERTa's 8 extra tokens (50,265 − 50,257) are exactly the special ones it needs and GPT-2 does not. <s> deserves special attention: it is the first token of every sequence and, because it sees everything (bidirectionally), it ends up working as a summary of the whole sentence. Classification comes out of it — and that is the subject of chapter 12.
.venv\Scripts\python.exe lab\10_roberta_mlm.py
The two RoBERTa models we downloaded are *-openai-detector: they were fine-tuned for classification, and in the process the MLM head was discarded and replaced with a binary classification head.
This means we cannot fill gaps with them — the necessary head is not in the weights. What we can do, and what the lab does: show the bidirectional body working (which is what matters conceptually), measure the difference in representation against GPT-2, and use the classification head they actually have. It is a real restriction of this set of models, and the lab records it instead of making the script look more complete than it is.
<s>/</s> wrapping the sentence and <mask> available in the vocabulary.The primary sources for the concepts in this chapter. The numbers in the text were measured in the lab; the ideas come from here.
Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.
You have seen the two branches separately. Chapter 11 puts them face to face
and shows that all the differences descend from a single line of code.