Encoder vs decoder: one line of code and every consequence that descends from it
The chapter that ties the whole course together, with the experimental proof and the decision table.
Landed here from LinkedIn? This is chapter 11 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.
A minus-infinity sign decides whether the model writes or understands. There is no free lunch. GPT-2 and RoBERTa have the same block, the same attention, the same tokenization algorithm, the same GELU, the same LayerNorm, and practically the same size: 124 million versus 125. The entire difference is one line, and a tree of consequences descends from it. The decoder trades understanding for generation. The encoder trades generation for understanding. And in 2026 it became common sense that decoders won. That is half true. The other half is that calling a 70-billion-parameter model to decide whether an e-mail is spam is waste by several orders of magnitude.
The three points
- The line: the causal mask. With it, the model only looks backward and can generate. Without it, it looks both ways and cannot.
- The experimental proof: the same sentence, both models, and the pronoun's attention changes in only one of them.
- To classify, search, extract: encoder. To write, converse: decoder. To translate and summarise: both together. The encoder did not die.
From the chapter, the beginning
One line of code, and all the consequences that descend from it. The chapter that ties the whole course together.
That's it. Everything else is the same or almost the same: the same Transformer block, the same attention equation, the same tokenization algorithm (byte-level BPE), the same GELU, the same LayerNorm, practically identical sizes (124 M vs 125 M, both with 12 layers, 768 dimensions and 12 heads).
gpt2 and roberta-base are structural twins that do opposite things. It is the perfect controlled experiment, and that is why studying them together teaches more than studying either one alone.
What the chapter covers · 7 sections · 6 min
- The line
- The experimental proof
- The tree of consequences
- The practical decision table
- The third family: encoder-decoder
- The lineage, on one page
- In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
- 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
- Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
- 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
- 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
- 03 · Attention: the single operation that replaced twenty years of recurrent networks
- 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
- 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
- 06 · What scale really buys: from 124 million to 1.5 billion, measured
- 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
- 08 · Perplexity: the one number that says what the model learned, without anecdotes
- 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
- 10 · RoBERTa: when the advance is training better what already existed
- 11 · Encoder vs decoder: one line of code and every consequence that descends from it
- 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
- 13 · Reading GPT-2's original code: 174 lines against everything you learned
- 14 · Serving models: from function call to HTTP server, and what production does on top
- 15 · The lab guide: every command, every script, and everything that was measured
- 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →