Rafhael Oliveira · IA

What scale really buys: from 124 million to 1.5 billion, measured

The complete assembly of GPT-2 and an honest measurement of what changes when the model grows.

1 min read· chapter: 8 min

Landed here from LinkedIn? This is chapter 06 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

GPT-2 large, with 774 million parameters, got the capital of France wrong. Medium, with 355 million, got it right. That contradicts the intuition that more parameters is always better, and it is a real result, reproducible on an ordinary machine. Two things are going on, and neither is 'the model is dumb'. One is about how the next word is chosen. The other is about what a single sample can measure. When you give three examples in the prompt, small gets it right 12 times more often. Prompt engineering was born here.

The three points

  1. The whole family uses the same block. What changes is how many and how big: 12 to 48 layers, 768 to 1600 dimensions.
  2. Scale is not monotonic on a single sample. Greedy takes the most likely and never backtracks; a tie on the first token decides everything.
  3. Three examples in the prompt multiply the probability of the right answer by 12 in small and by 10 in XL. Few-shot works, but GPT-2 is too small to be reliable.

From the chapter, the beginning

The full assembly of the pieces — and an honest measurement of what changes from 124 million to 1.5 billion parameters.

For a 5-token input, GPT-2 returns 5 predictions, one per position: "what comes after The", "what comes after The capital", and so on. All 5 are computed, always.

In training, all 5 are used — that is what makes pre-training so efficient (chapter 08). In generation, 4 are thrown away and only the last one matters. That "waste" is exactly what the KV cache recovers.

Keep reading chapter 06 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 5 sections · 8 min
  1. The whole model, top to bottom
  2. The whole family in one table
  3. What scale really buys
  4. The cost of running, measured
  5. In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →