Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
Why this chapter exists, the five rules, and the mistakes that only appeared when someone asked 'compared to what?'.
Landed here from LinkedIn? This is chapter 06 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.
Half of the best decisions in this project came from finding out we were wrong. The cosine regulariser exists because a semantic search failed. The floor metric exists because someone asked 'compared to what?'. Recording only what worked would distort the reading of everything else. This chapter records the five rules that make a comparison legitimate and the complete catalogue of mistakes, including the ones the CPU test does not catch. Whoever repeats the project will err differently. The kind of error repeats.
The three points
- Rule 1: a comparison is only valid if the only thing different is what is being tested. Frozen data, same steps, same seed.
- The floor rule: every gain is measured against the simplest possible thing. Copying the state, a random direction, the raw mean.
- Four mistakes in detail, each with what it taught. And the mistakes the CPU test does not catch.
From the chapter, the beginning
The rules that make a comparison legitimate, and the complete catalog of this project's mistakes — including the ones that only surfaced because someone asked "compared to what?".
Recording only what worked would distort the reading of everything that follows. And there is also a practical reason: half of the best decisions in this project came from discovering we were wrong. The cosine regularizer exists because a semantic search failed. The floor metric for the compass (the first version of the rudder, which only predicted the direction of the next state) exists because a pretty measurement turned out to be empty.
This chapter records the rules and the mistakes. Whoever repeats the project will err differently — but the kind of error repeats.
What the chapter covers · 6 sections · 8 min
- Why this chapter exists
- The five rules
- The catalog
- Four mistakes in detail
- The mistakes a CPU test does not catch
- The principles, distilled
All chapters of the experiment · Antivision · Latent Rehearsal
- 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
- 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
- 02 · Two architectures, one file: the fair duel between 2019 and what came after
- 03 · The geometry of the embedding: training starts with a perfect space and ruins it
- 04 · The lineage of collapse: five answers from computer vision, each measured on text
- 05 · The rudder and the world model: three versions of the same question, each measured against the last
- 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
- 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
- 08 · The workbench: serve, x-ray and train from the browser
- 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
- 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
- 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
- 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
- 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
- 14 · Consulted memory: a bank of abstract states read by attention
- 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
- 16 · The minimum was off the list: the day I measured my own search by energy
- 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
- 18 · Searching better does not fix the energy: three searches, the same text
- 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
- 20 · What changed on 17/09: two new chapters and two corrections
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →