Rafhael Oliveira · IA

The ceiling of this path: LeCun's argument, read from inside GPT-2

The finite-dictionary trick, the 10^14 bytes arithmetic, and where prediction happens. The close of the course, and the bridge to the experiment.

1 min read· chapter: 10 min

Landed here from LinkedIn? This is chapter 16 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

Every text humanity has written fits inside what a four-year-old has already seen. Both land around 10^14 bytes: four years of a small child, through vision and touch, and twenty to thirty trillion tokens at three bytes each. The arithmetic is Yann LeCun's, and it is the strongest argument about the limit of the path GPT-2 opened. Fifteen chapters took that model apart down to the last line. This one answers the question they left open: how far the path of LLMs goes. Not for want of data or parameters. The shape of the problem.

The three points

  1. Softmax over the vocabulary exists only because the dictionary is finite: GPT-2's central trick does not apply to a continuous world.
  2. Perplexity is honest about prediction and blind to planning — the ruler used across the whole course cannot see what is missing.
  3. Predicting a symbol versus predicting a representation is the causal-mask choice at another scale: where prediction happens.

From the chapter, the beginning

Where, according to Yann LeCun, the path GPT-2 opened reaches its limit — and why that limit is a research question, not a verdict against the model. The full argument, read from inside the model you have just taken apart.

Chapter 15 ends on a question it does not answer: how far does the path GPT-2 opened go? Fifteen chapters took that model apart down to the last line, on an ordinary machine, with no GPU. You saw text become numbers, saw attention replace twenty years of recurrent networks, saw what scale buys and what it does not, and measured perplexity instead of trading anecdotes.

Now the part that looks ahead. All of it works — and it was the foundation for almost everything that came after. Even so, there is a serious argument that this path, on its own, has a ceiling. The argument is Yann LeCun's, and it is not about engineering: not "needs more data", "needs more parameters", "needs more context". It is about the shape of the problem the model solves.

This chapter is that argument, piece by piece, each one pointing back at the piece you already opened in the course. The ideas are LeCun's and are quoted as his. Reading it in terms of GPT-2 is our part.

Keep reading chapter 16 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 9 sections · 10 min
  1. The question the course left open
  2. The trick only works because the dictionary is finite
  3. The arithmetic that equates a four-year-old with all of text
  4. Moravec's paradox, again
  5. Where prediction happens
  6. It is not a function, it is a relation
  7. Planning is not generating faster
  8. What this changes about what you measured
  9. What has been tested, and what failed
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →