The rudder and the world model: three versions of the same question, each measured against the last
The one that only pointed, the one that points better, and the one that imagines. With what each cost and bought.
Landed here from LinkedIn? This is chapter 05 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.
The question is a single one: does the model know where it is going before it goes? A predictor that does not know which action was taken can only predict the average of the possible futures. It is the blur LeCun talks about. It is what the first version measured: cosine 0.534 with the target, against 0.46 from simply copying the state. A gain of 0.07, and it cost quality. The second version, which predicts the next hidden state, was the only piece in the whole project that improved the generator. The third receives the action: 'if I emit this token, where do I end up?'. It predicts with cosine 0.93. And it cost the generator up to 9.59%.
The three points
- v1, shallow rudder: +1.41% cost, 0.07 gain over the floor. v2, deep rudder: −0.90%, improves the generator, perplexity 185.55, the best of all.
- v3, world model: takes the token as the action and predicts the next state. Error 0.150 against floor 0.948; cosine 0.93. Cost +9.59% with weight 1.0; +3.25% with weight 0.1.
- What would invalidate v3 was written before the runs: zero gain, the control without SigReg winning, or a cost bigger than what the deep rudder gives back.
From the chapter, the beginning
The central piece in three versions — the one that only pointed, the one that points better, and the one that imagines — with what each one cost and bought. The empty cells wait for runs that are in the queue; none will be filled in from the head.
The question is a single one: does the model know where it is going before it goes? Each version answers in its own way, and each was measured against the previous one under the same protocol (chapter 06): same frozen data, 700 steps, seed 1337, best checkpoint, control = cosine 1.0 with validation 5.2706.
A predictor that does not know which action was taken can only predict the average of the possible futures. It is the blur of the talk — "the system predicts the average of everything that can happen" — and it is what v1 measured: cosine 0.534 with the target, against 0.46 for copying the state. Gain of 0.07. The world model receives the action: "if I emit this token, where will I end up?". It is that question, and not the previous one, that makes planning possible.
What the chapter covers · 5 sections · 18 min
- Three versions of the same question
- What v1 and v2 measured
- The world model, piece by piece
- Inference: how the rudder becomes a rudder
- Correcting the map
All chapters of the experiment · Antivision · Latent Rehearsal
- 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
- 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
- 02 · Two architectures, one file: the fair duel between 2019 and what came after
- 03 · The geometry of the embedding: training starts with a perfect space and ruins it
- 04 · The lineage of collapse: five answers from computer vision, each measured on text
- 05 · The rudder and the world model: three versions of the same question, each measured against the last
- 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
- 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
- 08 · The workbench: serve, x-ray and train from the browser
- 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
- 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
- 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
- 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
- 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
- 14 · Consulted memory: a bank of abstract states read by attention
- 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
- 16 · The minimum was off the list: the day I measured my own search by energy
- 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
- 18 · Searching better does not fix the energy: three searches, the same text
- 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
- 20 · What changed on 17/09: two new chapters and two corrections
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →