Practically the same size, less than half the perplexity. Four decisions explain the difference.
openai-gpt (GPT-1, June 2018) has 117 million parameters. gpt2 small (February 2019) has 124 million. Six percent difference — practically nothing.
And yet, measured in lab 08 on the same paragraph:
This is rare and valuable. Many advances in language models came together with an increase in scale, which makes it hard to separate the effect of each change. Here scale is essentially controlled, and we can attribute the gain to specific design decisions. There are four.
| GPT-1 | GPT-2 | |
|---|---|---|
| Corpus | BooksCorpus | WebText |
| Content | ~7,000 unpublished fiction books | ~8 million web pages |
| Size | ~1 GB | ~40 GB |
| Curation | whole books | Reddit links with ≥3 karma |
| Diversity | low — almost only literary narrative | high — news, forums, code, documentation, recipes |
Forty times more data, and — more importantly — much more varied data. BooksCorpus teaches how to write fiction prose. WebText teaches how to write fiction prose, news, tutorials, forum comments, READMEs and code.
This shows up glaringly in the lab. Ask GPT-1 for code and it returns narrative. It has never seen a block brace in its life.
How do you collect 40 GB of quality text from the internet without hiring raters? OpenAI used an existing clue: if a human found the link good enough to post, and other humans upvoted it, it is probably not garbage. Three karma votes as a quality filter — cheap, scalable and surprisingly effective.
It is worth noting the bias this bakes in: WebText is, by construction, a portrait of the interests of Reddit users in 2017. Mostly in English, mostly American, with the topics and vocabulary of that community. When you observe GPT-2 being bad at Portuguese (lab 08: PPL 413), remember that the cause starts here.
| GPT-1 | GPT-2 | |
|---|---|---|
| Algorithm | BPE over characters, with preprocessing by spaCy and ftfy | BPE over bytes, no preprocessing |
| Vocabulary | 40,478 | 50,257 |
| Case | everything lowercased | preserved |
| Unknowns | <unk> token: information lost | impossible — everything is a byte |
| Punctuation | separated by spaces | preserved as is |
GPT-1 lowercases everything. That discards real information: in English, the capital letter distinguishes a proper noun from a common noun (Bill vs bill, Apple vs apple). And, on the output side, the model can never write a proper noun correctly.
You see this immediately in the lab test. Same prompt, GPT-1 returns:
gpt1: "the capital of france is a great city , and the city
is a great city . it is"
gpt2: "The capital of France is the capital of the French
Republic, and the capital of the"
Note three things in GPT-1's output: france in lowercase, the space before the comma (a residue of the preprocessing that separated punctuation), and the vague novel-prose structure — the echo of BooksCorpus.
You already know this one from chapter 05. GPT-1 uses Post-LN (LN(x + f(x))); GPT-2 uses Pre-LN (x + f(LN(x))), and adds the final ln_f.
In GPT-2 small, with 12 layers, the gain in quality is modest. The decisive gain is in feasibility: in lab 05 you measure that, in a 40-layer stack, Pre-LN delivers a gradient one and a half million times larger at the first layer.
Without this change, gpt2-xl with 48 layers simply would not exist — not in 2019, not without a long battle against numerical instability. Post-LN works well up to ~20 layers and then becomes a problem.
This is the most conceptual change, and perhaps the most influential of the four.
GPT-1 was presented as a two-stage method: pretrain on a large corpus, then fine-tune with labeled data for each specific task. The paper is called "Improving Language Understanding by Generative Pre-Training" — pretraining is the means; fine-tuning is the end.
GPT-2 proposed something more radical, right in the title: "Language Models are Unsupervised Multitask Learners". The thesis: if the corpus is large and varied enough, the model learns the tasks on its own, because examples of translation, summarization and question answering are already naturally present in internet text. No fine-tuning needed — just ask the right way.
It was here. The idea that you specify the task inside the input text, instead of training a separate model for it, is born in the GPT-2 paper. When you write "Translate to French:" at the start of a prompt, you are applying that thesis directly.
In GPT-2 the idea worked poorly (you measured it in lab 06: few-shot helps, but the model is too small to be reliable). In GPT-3, with 175 billion parameters, it worked so well that it made fine-tuning dispensable for most tasks — and changed the industry.
| GPT-1 (2018) | GPT-2 (2019) | |
|---|---|---|
| Parameters | 117 M | 124 M — 1,558 M |
| Layers | 12 | 12 — 48 |
| dmodel | 768 | 768 — 1600 |
| Context | 512 | 1024 |
| Vocabulary | 40,478 | 50,257 |
| Tokenizer | character BPE, lowercase | byte BPE, case preserved |
| Corpus | BooksCorpus, 1 GB | WebText, 40 GB |
| LayerNorm | Post-LN | Pre-LN + ln_f |
| Activation | GELU | GELU |
| Attention | causal, multi-head | causal, multi-head |
| Weight tying | yes | yes |
| Intended use | pretraining + per-task fine-tuning | zero-shot / few-shot in the prompt |
| PPL (our test) | 46.07 | 21.70 |
.venv\Scripts\python.exe lab\09_gpt1_vs_gpt2.py
<unk> token in action: gives GPT-1 an emoji and a Japanese word, and shows the information being lost.<unk> and GPT-2 does not. For a well-behaved English-only corpus, does that matter much? Where does the difference really show up?The primary sources for the concepts in this chapter. The numbers in the text were measured in the lab; the ideas come from here.
Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.
We have closed the decoder branch: GPT-1 → GPT-2 small → medium → large → XL.
Chapter 10 crosses over to the other branch of the tree, and everything you learned will be reused — with one line inverted.