RoBERTa: when the advance is training better what already existed

Four months after GPT-1, Google took the same block and made the opposite decision. Without the causal mask, every token sees the whole sentence, past and future. The representation of 'bank' in 'I sat on the bank of the river' is built with 'river' available from the first block. That is BERT. But bidirectionality creates a training problem: if the model sees everything, predicting the next word becomes cheating. The fix was to hide words. And a year later, Facebook showed BERT was undertrained, and that sometimes the advance is training better what already existed.

The three points

1. MLM: hide 15% of the tokens and ask the model to guess them using both sides. It is the objective that makes sense without a causal mask.
2. RoBERTa did not change the architecture. It changed data, training time, batch size and the masking, which became dynamic.
3. The structural differences you will touch: tokenizer, special tokens, and the absence of generation.

From the chapter, the beginning

Masked Language Modeling, dynamic masking, and the lesson that sometimes the advance is training better what already existed.

Without the mask, each token sees the whole sentence — past and future. The representation of "banco" in "sentei no banco da praça" (Portuguese: "I sat on the bench in the square") is now built with "praça" available from the very first block.

The gain in comprehension is enormous and immediate. BERT broke records on eleven tasks at once. But there is a price, and it is fatal for one specific thing.