What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
The talk on world models and the interview, with the project file where each idea enters, what was already right and what was corrected.
Landed here from LinkedIn? This is chapter 10 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.
The video that started it all: Yann LeCun explaining what LLMs still lack to understand the world. We read it concept by concept and applied it to text. Two sources: the talk on world models, technical, and the interview, less technical and more precise on three points. A viewer asked: 'are you proposing an alternative to the Transformer?'. The answer is direct: no. JEPA is a macro-architecture, an arrangement of modules, and the modules can be Transformers. What it replaces is the objective: predicting symbols. Reasoning is searching in mental space, not in output space. Emotion is anticipation of the outcome. Every concept in those talks has a file in our project where it applies, and several were corrected because of them.
The three points
- JEPA is not an alternative to the Transformer. It is a macro-architecture, and the Transformer is the encoder inside it.
- 'Predicting the world is impossible' holds where there is unpredictable detail. In text, the discrete action already removes what JEPA removes in video. What remains is the dynamics of the state.
- Reasoning is searching in mental space. Emotion is anticipation. Persistent memory is a requirement. All three became pieces, and all three are measured.
The video
Yann LeCun's talk that this chapter reads concept by concept. Watch before or after; the chapter cites the passages with the minute.
From the chapter, the beginning
Two transcripts — the talk on world models and the interview — read concept by concept, with the project file where each one applies, what was already right and what was corrected because of them.
The first source is Yann LeCun's talk on world models, the technical one: the world model, inference by energy, collapse and how to avoid it, SigReg, hierarchy, DINO as encoder, V-JEPA and expectation violation. The second is an interview with LeCun, less technical and more precise on three points that the talk leaves implicit: what JEPA is in relation to the Transformer, why predicting the world is impossible (and in which space it stops being so), and what reasoning is. This chapter exists because these three points change the reading of what we built.
What the chapter covers · 9 sections · 26 min
- The two sources
- JEPA is not an alternative to the Transformer
- "Predicting the world is impossible" — where, exactly
- Reasoning is searching in mental space, not in output space
- Emotion is anticipation of the outcome
- Persistent memory
- Three more pieces, from the second reading
- The map: concept → file
- The searches in the text
All chapters of the experiment · Antivision · Latent Rehearsal
- 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
- 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
- 02 · Two architectures, one file: the fair duel between 2019 and what came after
- 03 · The geometry of the embedding: training starts with a perfect space and ruins it
- 04 · The lineage of collapse: five answers from computer vision, each measured on text
- 05 · The rudder and the world model: three versions of the same question, each measured against the last
- 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
- 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
- 08 · The workbench: serve, x-ray and train from the browser
- 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
- 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
- 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
- 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
- 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
- 14 · Consulted memory: a bank of abstract states read by attention
- 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
- 16 · The minimum was off the list: the day I measured my own search by energy
- 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
- 18 · Searching better does not fix the energy: three searches, the same text
- 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
- 20 · What changed on 17/09: two new chapters and two corrections
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →