Rafhael Oliveira · IA

Perplexity: the one number that says what the model learned, without anecdotes

The loss, the perplexity, and the two lab results worth the whole chapter.

1 min read· chapter: 10 min

Landed here from LinkedIn? This is chapter 08 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

GPT-2 was never taught to answer questions. It was trained on a single task, billions of times. Predict the next word. That is all. And to predict the next word well in any text on the internet, the model is forced to learn, as a side effect, spelling, grammar, facts and the rudiments of reasoning. There is no shortcut: to complete 'The capital of Japan is', you have to know. There is a single number that measures how much it learned. On the same paragraph, GPT-1 scores 46. GPT-2 XL scores 12.9. In Portuguese, GPT-2 scores 413.

The three points

  1. Perplexity is the effective number of options the model is torn between at each word. 1 is certainty; 50,257 is having learned nothing.
  2. It is only comparable within the same tokenizer. Comparing GPT-2 with Llama by perplexity is comparing different things.
  3. Perplexity measures frustrated expectation, not disorder: random characters score 469, shuffled English scores 3,874. The model suffers more from the almost-right than from chaos.
Keep reading chapter 08 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 7 sections · 10 min
  1. The task: predict the next word. That's all.
  2. The loss: cross-entropy
  3. Perplexity: the loss in human units
  4. How GPT-2 was trained, in practice
  5. The code
  6. In the lab
  7. Two lab results worth the whole chapter
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →