Reading GPT-2's original code: 174 lines against everything you learned
The primary source, the whole block in the original, and what the gpt-3 repository really contains.
Landed here from LinkedIn? This is chapter 13 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.
The whole of GPT-2 fits in 174 lines of TensorFlow. And you can read every one of them. The primary sources are public. They are not for running: they use a module removed from TensorFlow in 2019 and do not support Python above 3.7. They are for reading. After twelve chapters, you have the vocabulary to understand every line, and reading the original has a value no second-hand explanation replaces. There are three findings in there that only appear to whoever reads the source. One of them is about the causal mask, and it is not how the tutorials teach it.
The three points
- The Transformer block in the original, side by side with the course's version. The same pieces, with the names OpenAI gave them.
- The causal mask in the original is built in a way the tutorials do not show.
- The gpt-3 repository has neither code nor weights. It has text samples and statistics. There is no 'downloading GPT-3'.
From the chapter, the beginning
The whole of GPT-2 fits in 174 lines of TensorFlow. Reading the primary source, line by line, against everything you have learned.
It uses tensorflow.contrib, a module removed in TensorFlow 2.0 (2019). And TF 1.15, the last version that has it, does not support Python above 3.7 — we are on 3.14. Making it run would require a separate Python 3.7 environment and a 2019 version of TF.
It is not worth it, and the reason is a good one: the weights are the same. The gpt2 you load with transformers is exactly the model in this repository, converted from a TensorFlow checkpoint to safetensors. You are already running the original GPT-2 — just in an implementation that works on your Python.
What the chapter covers · 7 sections · 6 min
- What is in repos/
- The whole block, in the original
- The causal mask, in the original
- Three findings from reading the original
- The tokenizer: GPT-3-Encoder
- The gpt-3 repository: data, not code
- In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
- 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
- Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
- 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
- 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
- 03 · Attention: the single operation that replaced twenty years of recurrent networks
- 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
- 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
- 06 · What scale really buys: from 124 million to 1.5 billion, measured
- 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
- 08 · Perplexity: the one number that says what the model learned, without anecdotes
- 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
- 10 · RoBERTa: when the advance is training better what already existed
- 11 · Encoder vs decoder: one line of code and every consequence that descends from it
- 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
- 13 · Reading GPT-2's original code: 174 lines against everything you learned
- 14 · Serving models: from function call to HTTP server, and what production does on top
- 15 · The lab guide: every command, every script, and everything that was measured
- 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →