Two claims from LeCun that became experiments: "combine perception with the contents of a memory" and "each level of abstraction predicts further than the one below". Both were measured, and both hold for text — at the honest scale of a 7.4 M model.
LeCun lists persistent memory, in both talks, as a requirement of an intelligent system, and the talk says where it enters: perception gives only the current state; the full state is perception combined with the contents of a memory. An ordinary language model has no such thing — it has the context window, and what leaves it ceases to exist.
One vector abstrair()) and folded into the memory:
The world model then receives
Every evaluation measures the world model three times on the same batch: with the right memory, with the memory of another sequence in the batch, and with no memory. The memory gain is
ganho_memoria = MSE(wrong memory) − MSE(right memory)
Zero means that
Three contiguous windows A, B, C. The world model predicts inside C, which it sees whole. The memory comes from different sources (model/memory.py):
| Memory source | MSE | gain vs other | What it tests |
|---|---|---|---|
| zero | 0.220 | −0.028 | no past: the worst of all |
| other (A+B from another text) | 0.193 | 0 | the control |
| recent (B only) | 0.173 | +0.019 | the immediate past |
| long (A+B) | 0.181 | +0.012 | the whole past: less than each half alone |
| old only (A only) | 0.174 | +0.019 | persistence: crossed a whole window and is worth as much as the recent one |
| floor: copying | 1.130 | — |
Run wm1n2mem-sz1 closed at 5.6801 (+7.77%, in the range of the other end-to-end runs), map gain +0.93, and ganho_memoria +0.014 on the training validation. In the three-window experiment, the order is the one the hypothesis asked for: the right memory predicts better than another text's (0.173 against 0.193, about 10% of the map's error), and another text's predicts better than none (0.220) — the vector helps as a bias, and helps more when it is the right past. The number that matters is the last row: the memory from A only, separated from C by a whole window of 256 tokens, is worth the same as the one from B (+0.019 against +0.019). What left the window did not cease to exist. It is persistence, not recency.
One surprise: "long" (A+B) yields less than each half alone (+0.012). Folding two blocks of 256 tokens into the same 256-dimensional vector dilutes; the memory as it stands is a summary with fixed capacity, and the excess costs. It is the argument for memory that is queried rather than summed (open question below).
"Long" ≈ "other" ≈ "zero": the memory carries nothing, the world model learned to ignore it. That is a possible result and not a bug: with a 256-token window, there may be no useful information left outside it in a validation text. If "recent" gains and "old only" does not, the memory is short-term — recency, not persistence.
Neither of the two invalidations happened: "long" sits between "recent" and "other", and "old only" ties with "recent". Honest scale: the gain is of the order of 10% of the map's error, with 32 sequences and one seed.
At generation, the memory works the same way: the prompt that does not fit in the window is folded in before generating, and each block that leaves the window during generation is folded in too. It was tested with an 81-token prompt in a 64-token window; the smoke-test number (two-step world model) gave every source tied at 1.46, as it should be on a map that has not yet learned anything.
The talk justifies the hierarchy with one sentence: "each level of abstraction allows longer-range predictions than the level below, by ignoring details". It is the reason level 2 exists, and it had never been measured here. The experiment (model/horizon.py):
abstrair(), against the same | Horizon (tokens) | MSE level 1 | floor | gain | cos level 1 | cos floor |
|---|---|---|---|---|---|
| 1 | 0.167 | 1.118 | +0.951 | 0.911 | 0.421 |
| 2 | 0.197 | 1.206 | +1.009 | 0.895 | 0.375 |
| 4 | 0.211 | 1.205 | +0.995 | 0.888 | 0.376 |
| 8 | 0.222 | 1.211 | +0.989 | 0.885 | 0.373 |
| Predictor | MSE | gain | cos |
|---|---|---|---|
| floor (copying | 0.171 | 0 | 0.810 |
| level 1 rolled out × 8, abstracted | 0.152 | +0.020 | 0.822 |
| level 2, one step | 0.094 | +0.078 | 0.891 |
Level 1 rolled out degrades slowly: from cosine 0.911 at one step to 0.885 at eight, while the identity stays at 0.37–0.42. Eight tokens ahead, without ever correcting the imagined state, the map is still at 0.885 of the real. On wm1n2mem-sz1, on the abstract yardstick, the same order: floor 0.174 (cos 0.81), level 1 rolled out × 8 0.288 (cos 0.65, worse than the floor), level 2 in one step 0.102 (cos 0.89). And on wm0.1n2-sz1-lat1o: floor 0.400, level 1 rolled out 0.319, level 2 0.220 (cos 0.74). And on the ζ with the channel closed (wm1n2z16zd0.5-sz1, ζ = 0): floor 0.195, level 1 rolled out 0.162, level 2 0.106 (cos 0.87). On the world model on top of the frozen DINO encoder (wm-sobre-dino, learned level-2 encoder without norm): floor 0.066, level 1 rolled out 0.045, level 2 0.038 (cos 0.44) — the learned encoder shrank again (the raw mean would give ~0.16), less than in the end-to-end one, and level 2 stays ahead. On five of five hierarchical models, level 2 predicts the next block better than level 1 rolled out.
| Predictor | MSE | gain | cos |
|---|---|---|---|
| floor (copying | 0.013 | 0 | 0.274 |
| level 1 rolled out × 8, abstracted | 0.0095 | +0.0035 | 0.194 |
| level 2, one step | 0.0068 | +0.0062 | 0.448 |
With the learned encoder, the block floor fell from 0.171 to 0.013 and the floor's cosine from 0.810 to 0.274: the abstract states shrank thirteen times and became almost orthogonal between consecutive blocks. Level 2 "gains" +0.006 over a floor of 0.013 — the same relative fraction as the raw mean (about 45%), on a scale where level 2's MSE is small by construction. It is the collapse disguised as accuracy that chapter 09 wrote down as a risk before the run: the encoder learns to send every block close to the same point, and the predictor "gets it right". SigReg on the level-1 states does not reach level 2 — the first fix tried was the same anti-collapse applied to the abstract states (--sigreg-s2 1, run wm1n2e-sz1-ss1). Result: the block floor rose from 0.013 to 0.068 (still below the raw mean's 0.160), level 2 still beats level 1 rolled out (0.033 against 0.050, gain +0.039 against +0.021), but consecutive blocks became almost orthogonal — floor cosine 0.07, against 0.81 on the raw mean — and the generator paid +11.32%, the worst cost of all the runs. Gaussianizing the abstract states with weight 1.0 prevents the shrinking by making each block independent of the previous one, which is the opposite of what a level-2 state should be. The second fix was scale-only (--world-enc2-norm: the encoder's output keeps the raw mean's norm, without forcing independence; run wm1n2en-sz1). Result: collapse through the other door. With the norm fixed, the encoder sent every block to the same direction — cosine 1.000 between consecutive blocks, block floor 0.009, level 2 "hitting" 0.005 — and the generator stayed at +6.22%, the cheapest of the learned encoders precisely because the level-2 loss became trivial. Three attempts, three shortcuts: shrinking (mean + correction), aligning (fixed norm), or losing continuity (SigReg). A learned level-2 encoder, trained only by predicting the next block, always finds the solution that makes the next block easy to predict — and the easiest one is the one that does not change. It is the talk's collapse in pure form, and the lesson is that the level-2 anti-collapse needs to fix scale and variance between blocks without destroying the correlation between neighbours (a variance term, like VICReg's, on the abstract states), which remains open. On this budget, the raw mean (wm1n2-sz1) is the honest abstraction, and it is the one that counts for the claim below.
On the abstract yardstick, level 2 in one step errs 0.094 against 0.152 for level 1 rolled out eight times and abstracted — gain +0.078 against +0.020 over the floor of copying the block, four times more. Abstraction predicts further. Level 1, even predicting each token with cosine 0.89, accumulates error along the block and reaches the next block barely better than the identity; level 2, which never saw the individual tokens, gets closer in a single step. It is exactly "each level ignores detail to predict further" — measured, in a 7.4 M model, with blocks of 8. And it is what makes level 2 worth having as a source of subgoals for level 1 (chapter 09): it knows where the next block will be better than level 1 can imagine.
If the gain of level 2 exceeds that of level 1 rolled out × 8, abstraction predicts further: the talk's claim holds for text in this model, and level 2 buys range, not just economy (one step instead of eight). If level 1 rolled out wins, level 2 is not abstracting anything useful — and the run with the learned level-2 encoder (wm1n2e-sz1) is the next one to look at. Level 1 rolled out should degrade with
abstrair(). It only makes sense after the second proves it predicts further.Chapter 12 measures the other end: with the map trained, which inference mode steers best — and at what price.