GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
The controlled experiment history handed us, and what it teaches about where the gain comes from.
Landed here from LinkedIn? This is chapter 09 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.
Do you know how GPT-1 became GPT-2? And why it happens? GPT-1 has 117 million parameters. GPT-2 small has 124 million. Six percent difference, practically nothing. And yet, measured on the same paragraph, perplexity drops from 46 to 21.7. Less than half. It was not size. It was four decisions, and the biggest of them is not about architecture. It is about where the text came from, and it involves three upvotes on Reddit.
The three points
- Change 1, the biggest: the corpus. From 1 GB of fiction books to 40 GB of web pages filtered by Reddit karma.
- Change 2: the tokenizer. From BPE over characters, all lowercase, to BPE over bytes, case preserved, no unknown token.
- Change 3: Pre-LN, the engineering change. Change 4: what did not change, but changed purpose. Prompt engineering was born here.
From the chapter, the beginning
Practically the same size, less than half the perplexity. Four decisions explain the difference.
openai-gpt (GPT-1, June 2018) has 117 million parameters. gpt2 small (February 2019) has 124 million. Six percent difference — practically nothing.
This is rare and valuable. Many advances in language models came together with an increase in scale, which makes it hard to separate the effect of each change. Here scale is essentially controlled, and we can attribute the gain to specific design decisions. There are four.
What the chapter covers · 7 sections · 7 min
- The controlled experiment history handed us
- Change 1 — the corpus (the biggest of all)
- Change 2 — the tokenizer
- Change 3 — Pre-LN (the engineering change)
- Change 4 — what did not change, but changed in purpose
- The final table
- In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
- 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
- Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
- 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
- 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
- 03 · Attention: the single operation that replaced twenty years of recurrent networks
- 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
- 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
- 06 · What scale really buys: from 124 million to 1.5 billion, measured
- 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
- 08 · Perplexity: the one number that says what the model learned, without anecdotes
- 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
- 10 · RoBERTa: when the advance is training better what already existed
- 11 · Encoder vs decoder: one line of code and every consequence that descends from it
- 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
- 13 · Reading GPT-2's original code: 174 lines against everything you learned
- 14 · Serving models: from function call to HTTP server, and what production does on top
- 15 · The lab guide: every command, every script, and everything that was measured
- 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →