What changed on 17/09: two new chapters and two corrections
An update note for anyone following the experiment: what closed, what was corrected, what changed in the code and what stayed open — with a link to each chapter.
Landed here from LinkedIn? This is chapter 20 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.
In one day the experiment gained two chapters and corrected two claims I had already published. Nine measurements closed on 17 September. Two of them overturned readings that were live: the exact search for the energy minimum does not improve the text (what decides is the energy, not where you search), and the world model does not start paying with more data — with the validation ruler fixed, it costs 1.77% on the mini model and 4.92% on pequeno. Two new chapters appeared, four older ones changed numbers, and no cell was left "pending" in the collection.
The three points
- New chapters: 18, which measures what the exact search for the lowest energy does to the text, and 19, which redoes the collection's strongest result with more seeds.
- Corrections: the world model's cost on the full corpus (chapter 15), the volume of low energy and the contrastive round (16), the 32-token blocks (13) and two rows of the concept → code map (10).
- In the code, all additive: fixed validation batches in training, energy search inside the policy's nucleus, and three new options for measuring the volume of low energy.
What the chapter covers · 4 sections · 4 min
- New chapters
- Chapters that changed
- What changed in the code
- What stayed open
All chapters of the experiment · Antivision · Latent Rehearsal
- 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
- 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
- 02 · Two architectures, one file: the fair duel between 2019 and what came after
- 03 · The geometry of the embedding: training starts with a perfect space and ruins it
- 04 · The lineage of collapse: five answers from computer vision, each measured on text
- 05 · The rudder and the world model: three versions of the same question, each measured against the last
- 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
- 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
- 08 · The workbench: serve, x-ray and train from the browser
- 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
- 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
- 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
- 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
- 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
- 14 · Consulted memory: a bank of abstract states read by attention
- 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
- 16 · The minimum was off the list: the day I measured my own search by energy
- 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
- 18 · Searching better does not fix the energy: three searches, the same text
- 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
- 20 · What changed on 17/09: two new chapters and two corrections
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →