Rafhael Oliveira · IA

Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa

Multi-head attention and the causal mask, with the complete code.

1 min read· chapter: 8 min

Landed here from LinkedIn? This is chapter 04 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

Attention is a fixed budget: every row sums to 1. If 70% goes to one word, 30% is left for everything else. But a word takes part in several relationships at once. 'Bit', in 'the big dog bit the mailman yesterday', needs to look at who bit, who was bitten and when. One head cannot cover it. The Transformer's answer was to divide, not multiply: twelve heads, each with its own budget. And right after that decision comes a line of code that is the only structural difference between GPT-2 and RoBERTa.

The three points

  1. Multi-head does not cost more: the 768 dimensions are split into 12 slices of 64. Each head sees one slice.
  2. The causal mask puts minus infinity on everything in the future. After the softmax, that becomes zero. The model cannot peek.
  3. Remove the mask and you have an encoder. Keep it and you have a decoder. Everything else is the same.

From the chapter, the beginning

Why twelve heads instead of one — and the single line of code that separates GPT from RoBERTa.

The attention from chapter 03 works, but it has a serious limitation: each row of the attention map sums to 1. That means attention is a fixed budget. If a token spends 70% looking at its syntactic antecedent, only 30% is left for everything else.

Keep reading chapter 04 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 5 sections · 8 min
  1. The problem with a single head
  2. How it works: divide, don't multiply
  3. The causal mask — the line that defines everything
  4. The complete code
  5. In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →