The model returns a distribution. Turning that into words is your decision — and it changes the result more than switching models does.
There is a very widespread misunderstanding: that "the model chooses the next word". It does not choose. The model returns a score for each of the 50,257 tokens in the vocabulary — and stops there. Choosing is the job of a separate algorithm, which runs afterwards and which you control.
These raw scores are called logits. They range from −∞ to +∞ and do not sum to anything in particular. To become probabilities, they pass through the softmax:
From here on, everything is decoding strategy. And the choice of strategy accounts for an enormous share of perceived quality — switching from gpt2 to gpt2-xl often changes the text less than switching from greedy to top-p.
It seems like the obvious choice — if we want the most likely text, we pick the most likely word at each step, right?
Wrong, and for two independent reasons.
The most likely word right now can lead into a dead end where every continuation is bad. A slightly less likely word now could open up a much better sequence. Greedy never discovers this, because it never backtracks. It is the same problem as any greedy algorithm.
This is the deep reason, and it was demonstrated in an important paper (Holtzman et al., 2020, "The Curious Case of Neural Text Degeneration"). The authors measured the probability a model assigns to human-written text, and compared it with the text the model itself generates in greedy mode.
Result: human text is far less likely. Humans constantly pick surprising words. A text where every word is as predictable as possible does not sound human — it sounds like a scratched record.
gpt2, greedy, prompt "The capital of France is": "the capital of the French Republic, and the capital of the French Republic, and the capital of the French Republic..." The loop is not a bug. It is the CORRECT behavior of greedy: once the sequence enters a high-probability cycle, each repetition makes the next one even more likely. It gets stuck.
Maximizing probability produces low-quality text. To sound human, you have to deliberately not always choose the most likely. Every technique in this chapter is a disciplined way of introducing surprise without introducing garbage.
The opposite extreme: draw the token according to the distribution. If " Paris" has 40% and " the" has 12%, you draw with those weights.
This fixes the repetition, but creates a new and treacherous problem: the long tail.
There are 50,257 tokens. Suppose the 50 thousand irrelevant tokens each have a minuscule probability of 0.001%. Individually, nothing. Summed: a 50% chance of a completely random token coming out.
In practice the distribution is more concentrated than that, but the effect is real: pure sampling produces text that goes along fine and suddenly plunges into a meaningless word — and, since the model is autoregressive, it now has to continue from that out-of-place word. One error contaminates everything that follows.
We need to cut the tail. The two ways of doing that are top-k and top-p.
Before the softmax, divide all the logits by a number
Notice that it is the same operation as the √
| T | Effect | Resulting distribution | When to use |
|---|---|---|---|
| 0.1 | almost greedy | 0.999 · 0.000 · 0.000 | factual answers, code |
| 0.7 | conservative | 0.822 · 0.061 · 0.015 | the sensible default |
| 1.0 | raw distribution | 0.665 · 0.090 · 0.033 | neutral |
| 1.5 | creative | 0.506 · 0.131 · 0.067 | fiction, brainstorming |
| 3.0 | chaotic | 0.332 · 0.171 · 0.122 | almost never |
temperature=0 is a division by zero and, mathematically, undefined. Libraries treat it as a special case: T=0 becomes pure greedy. In transformers, the correct way is do_sample=False. Passing temperature=0 with do_sample=True either raises an error or is silently ignored, depending on the version.
Keep only the
k = 5, prompt "The capital of France is":
kept: ' the' 0.28 | ' Paris' 0.19 | ' a' 0.09 |
' now' 0.06 | ' one' 0.04
discarded: the other 50,252 tokens
renormalize to sum to 1 and draw.
Simple and effective. But it has a conceptual flaw:
"A capital da França é" ("The capital of France is"), the distribution is very concentrated — maybe 2 tokens are plausible. With "Ele abriu a porta e viu" ("He opened the door and saw"), almost anything is plausible. With The elegant solution, available as a parameter in the main generation libraries and APIs. Instead of fixing how many tokens to keep, fix how much cumulative probability to keep:
gpt2| Context | 1st token | P(1st) | Nucleus p=0.9 |
|---|---|---|---|
"The United States of" | " America" | 0.966 | 1 |
"It was the best of" | " times" | 0.597 | 45 |
"The capital of France is" | " the" | 0.085 | 1,503 |
"He opened the door and saw" | " a" | 0.161 | 1,648 |
"Yesterday I went to the" | " store" | 0.023 | 5,548 |
Read the first and last rows together. In "The United States of" the model is 96.6% certain — the nucleus is a single token, and there really should be nothing to draw. In "Yesterday I went to the" it has no idea (the favorite has 2.3%) — and the nucleus opens up to 5,548 tokens, because anything is plausible there.
A fixed top_k=50 would treat the two situations the same: it would leave 49 bad candidates in the first and amputate the variety in the second. Top-p handles both with the same parameter, with no manual tuning. That is the whole insight.
Notice that "The capital of France is" lands in the doubt range, with a nucleus of 1,503 tokens and the favorite worth only 8.5%. gpt2 small is genuinely not confident about this question — which confirms, by a completely different route, what the logit lens (chapter 05) and the scale test (chapter 06) had already shown. Three independent measurements, the same conclusion. That is how you gain confidence in a result.
The repetition_penalty divides the logit of any token already used, discouraging repetitions. Values between 1.1 and 1.2 help a lot in small models. Above 1.3, the text gets strange — the model starts avoiding words that need to repeat, such as articles and prepositions.
| Goal | Configuration | Why |
|---|---|---|
| Factual / code | do_sample=False | deterministic and reproducible; repetition is an acceptable price |
| General use | temperature=0.7, top_p=0.9 | a common combination; balances coherence and naturalness |
| Creative | temperature=1.0, top_p=0.95 | more variety without opening the long tail |
| Small GPT-2 | temperature=0.8, top_p=0.9, | the small repeats a lot; the penalty compensates |
| Reproducing a result | any + fixed seed | sampling with a fixed seed is deterministic |
Libraries usually apply both in sequence (first top-k, then top-p). If you leave top_k=50 on together with top_p=0.95, top-k has already cut to 50 before top-p gets a chance to decide. In open contexts, where the nucleus would need 200 tokens, you never get there. If you want real top-p, turn top-k off (top_k=0).
Beam search keeps
That is why beam search is excellent for translation and summarization — tasks where there is one right answer and you want to find it — and bad for open-ended generation, where the goal is to sound natural. Do not use it to chat with GPT-2.
.venv\Scripts\python.exe lab\07_decoding.py
temperature with low top_p a strange combination? What does one do that the other undoes?The primary sources for the concepts in this chapter. The numbers in the text were measured in the lab; the ideas come from here.
Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.
We now know how the model predicts and how we choose among the predictions.
Chapter 08 goes back one step: what exactly it learned, and how that is measured with a single number.