Where, according to Yann LeCun, the path GPT-2 opened reaches its limit — and why that limit is a research question, not a verdict against the model. The full argument, read from inside the model you have just taken apart.
Chapter 15 ends on a question it does not answer: how far does the path GPT-2 opened go? Fifteen chapters took that model apart down to the last line, on an ordinary machine, with no GPU. You saw text become numbers, saw attention replace twenty years of recurrent networks, saw what scale buys and what it does not, and measured perplexity instead of trading anecdotes.
Now the part that looks ahead. All of it works — and it was the foundation for almost everything that came after. Even so, there is a serious argument that this path, on its own, has a ceiling. The argument is Yann LeCun's, and it is not about engineering: not "needs more data", "needs more parameters", "needs more context". It is about the shape of the problem the model solves.
This chapter is that argument, piece by piece, each one pointing back at the piece you already opened in the course. The ideas are LeCun's and are quoted as his. Reading it in terms of GPT-2 is our part.
Go back to chapter 07. The model produces a vector of logits, one number per vocabulary token, and softmax turns that into a probability distribution. That is how you sampled text with temperature, top-k and top-p.
LeCun points out that this whole gesture rests on one condition: the vocabulary is finite. In his words, it is easy because there is only a finite number of words, so you can get the system to produce a probability distribution over all possible words in the dictionary. You can never train a system to predict exactly which word comes next — but you can train it to produce a score for every word in a dictionary.
Notice what that grants and what it charges. It grants a tractable output: 50,257 numbers in GPT-2, and done. It charges that the world must fit in a dictionary. For text, it fits. For video, for sound, for the physical world, it does not — there is no "dictionary of all possible frames". The central trick of GPT-2, the one you watched work in lab 07, is a trick that only applies to discrete symbols.
This is the hardest number in the talk, and it is easy to check.
LeCun estimates that a four-year-old has already taken in, through vision — and probably touch as well — something on the order of 1014 bytes. On the other side, he runs the arithmetic for the models: twenty or thirty trillion tokens at about three bytes each also comes to roughly 1014 bytes. A one followed by fourteen zeros, in both cases.
| Source | Volume | Over how long |
|---|---|---|
| A four-year-old, through vision and touch | ~1014 bytes | four years awake |
| 20 to 30 trillion tokens, ~3 bytes each | ~1014 bytes | essentially all public text |
Same order of magnitude. On one side, four years of a small child's life. On the other, practically everything humanity has written and digitised. And the child comes out of those four years knowing that objects fall, that a glass pushed off a table breaks, that a hidden face still exists. The model comes out of the same 1014 bytes able to write about it.
LeCun's conclusion is not that text is useless. It is that text is too narrow a channel to carry what he calls physical common sense. If all the text in the world fits inside what a child sees before learning to read, the bottleneck is not the amount of text.
LeCun returns to Moravec's paradox more than once: what is simple for people is hard for computers, and what is complicated for people — chess, symbolic integrals, theorems — comes relatively cheap on a machine.
The form he gives it is useful here: the space of discrete objects and symbols is easily manipulated by computers; the real world is not. And the techniques that work in one case do not work in the other. A cat or a dog learns about gravity in a few months. No current system comes close.
You measured that paradox without noticing, in chapter 06. Going from 124 million to 1.5 billion parameters bought real gains in perplexity — and GPT-2 large, at 774 million, still got the capital of France wrong where medium, at 355 million, got it right. Scale reliably buys fluency; it does not reliably buy a world model.
Here is the technical core of the argument, and it bears directly on the architecture you assembled in chapter 05.
LeCun describes what happens when you try GPT-2's trick on video: show the first segment and ask the system to predict what comes next at the pixel level. It does not really work, and — this matters most — the representations that come out of the system are not particularly good. The effort of predicting every detail of the output spends the model's capacity on what is unpredictable by nature.
His proposal is to move where the prediction happens. In the architecture he calls JEPA — joint embedding predictive architecture — you observe the input and the target, encode both, and the prediction takes place in that representation space, not in output space. He notes that the best image and video representation systems trained by self-supervised learning all use joint embedding.
Translated into what you took apart: GPT-2 predicts the next symbol. The alternative predicts the next state, in a learned space, and only then decides the symbol. Chapter 04 showed that a single line of code — the causal mask — separates GPT from RoBERTa. The distance between predicting a symbol and predicting a representation is larger than one line, but it is the same kind of thing: a choice of where prediction happens.
One formal detail holds up the rest. LeCun insists that, given an X, there is no single Y. The dependency between them is a relation, a mapping of some kind — but not a function.
You can see it in your own lab 07. For "the cat sat on the", there is no correct continuation; there are many. Softmax handles this by spreading probability, and sampling picks one. It works for text because the set is finite and enumerable. When Y is continuous — a video frame, a state of the world — spreading probability over every possible output stops being feasible.
The way out he has argued for over some twenty years is energy-based models: instead of a normalised distribution over outputs, a function that assigns low energy to compatible pairs and high energy to incompatible ones. And he ties it straight to reasoning: reasoning and planning, for him, proceed by energy minimisation rather than forward propagation — and for that to work you need a world model.
That sentence is the key to what follows. The whole GPT-2 you assembled is forward propagation: token in, token out, one pass. At no point in the lab does the model compare two possible continuations before emitting.
LeCun describes the missing loop: you feed an internal world model a proposal for a sequence of actions; the model predicts the outcome of that sequence; and the outcome goes to an objective that measures how far the task has been accomplished. The choice comes out of that comparison.
And he insists it has to be hierarchical. His example is domestic: there is no way to plan a whole trip to Paris in terms of muscle actions ten milliseconds at a time. Being in New York, going to the airport decomposes into going down to the street, hailing a taxi, and so on. Planning happens at levels of abstraction.
Set that against what you measured in chapter 08. Perplexity scores the quality of next-token prediction, one at a time. It is an honest metric for what the model does — and for that very reason it cannot see planning. A model that imagines three continuations and picks the best gains nothing in perplexity for it. The ruler used throughout this course is blind to the capability LeCun says is missing.
Nothing the course measured becomes wrong. What changes is the reading.
| What the course showed | What the argument adds |
|---|---|
| Tokenisation is a compression choice (ch. 01) | it is also imposing a finite dictionary on the world |
| Softmax gives a distribution over the vocabulary (ch. 07) | only possible because the vocabulary is enumerable |
| Scale buys perplexity (ch. 06) | it buys fluency, not physical common sense |
| Perplexity is the honest number (ch. 08) | honest about prediction, blind to planning |
| One line separates encoder from decoder (ch. 04, 11) | both predict symbols; the alternative predicts representations |
The course still stands in full as a description of how an LLM works inside. What it does not try to do is settle the right way to build intelligence — that question is still open, and it is exactly what the next experiment looks into.
Here the close of this course becomes the start of the other one, and it is worth being transparent about the results.
The Antivision experiment takes these ideas to ground the talk does not cover: text, with measurement. What it builds is an ELLM — a language model that predicts in latent space and decides by energy. Every idea becomes a training run against the same control, with the same data, the same steps and the same seed. So far that is 43 runs measured and 39 complete duels. Of those, 3 improved the generator and 33 cost it — most of the ideas, implemented the way we implemented them, made the model worse.
Those numbers are not a judgement on LeCun's ideas. It says only that the distance between a correct argument about the shape of the problem and a measured gain in text is large, and that it gets crossed run by run, publishing what fails as prominently as what works.
The sources of the ideas read in this chapter. The numbers quoted, such as the 1014 bytes arithmetic, are Yann LeCun's and come from the talk.
Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.
The course ends here, grateful to the model that taught all of this. The next experiment picks up from it, in public.