Rafhael Oliveira · IA

Temperature, top-k and top-p: the decision that changes the text more than switching models

The model returns a distribution. Turning that into words is your choice.

1 min read· chapter: 9 min

Landed here from LinkedIn? This is chapter 07 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

The model does not choose the next word. You do. There is a widespread misunderstanding. The model returns a score for each of the 50,257 tokens in the vocabulary and stops there. Choosing is the job of a separate algorithm that runs afterwards and that you control. Temperature, top-k, top-p: three knobs that change the result more than switching models. And measured on GPT-2 small, 'The capital of France is' has 1,503 tokens in its nucleus of doubt, with the favourite worth only 8.5%.

The three points

  1. Greedy always takes the most likely and falls into repetition. Pure sampling takes anything and falls into nonsense. Everything else is the middle ground.
  2. Temperature is the boldness knob: it flattens or sharpens the distribution before sampling. Top-k cuts at a fixed count; top-p cuts where the probability mass says.
  3. Beam search exists and is almost never used in open generation. The post explains why.
Keep reading chapter 07 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 9 sections · 9 min
  1. What the model really returns
  2. Greedy: always the most likely
  3. Pure sampling, and why it fails too
  4. Temperature: the boldness knob
  5. Top-k: fixed cutoff
  6. Top-p (nucleus sampling): adaptive cutoff
  7. Repetition penalty, and the ready-made recipes
  8. Beam search — and why it is not used here
  9. In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →