Rafhael Oliveira · IA

The lab guide: every command, every script, and everything that was measured

The reference chapter to come back to later. And the bridge to what comes next.

1 min read· chapter: 6 min

Landed here from LinkedIn? This is chapter 15 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

GPT-2 opened the path behind every LLM you use today. How far does that path go? Fifteen chapters took GPT-2 apart down to the last line. All measured on an ordinary machine, no GPU: perplexities, attentions, decoding, scale. And at the end of the road a question remains that the course does not answer, because it belongs to another project. GPT-2 chooses the next token and finds out afterwards where it ended up. Every LLM does that. Yann LeCun argues that this path, on its own, has a ceiling. We decided to test his proposal, on text, with numbers.

The three points

  1. Chapter-to-lab index: which script proves which concept.
  2. The table of everything measured on this machine, in one place.
  3. Where to go next: the Antivision series, where the model imagines before it chooses.

From the chapter, the beginning

Every command, every script, every file. The reference chapter to come back to later — and the summary of everything that was measured.

The lab scripts, the course server, the dependency list and the installation instructions are in the course's public repository: github.com/Rafhael-Oliveira-IA/conceitos-gpt. If you do not use git, you can download everything as a .zip file. The models are not in the repository, because they add up to 13.2 GB: the tools/download_models.py script downloads all seven straight from Hugging Face.

Keep reading chapter 15 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 7 sections · 6 min
  1. Start here
  2. Index: chapter → lab
  3. File layout
  4. The model aliases
  5. Everything that was measured on this machine
  6. Common problems
  7. Where to go next
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →