What we are building, why it is not a copy of GPT-2, and what this project's own contribution is — with what has already been measured, including what went wrong.
The previous course took apart models that already existed. This one builds one. The difference is not size — our model is smaller than any of those. The difference is that here every decision was measured, and several follow paths the field found after 2019, on the foundation GPT-2 left.
What it builds has a name: an ELLM, energy-based latent predictive language model — a language model that predicts in latent space and decides by energy. It keeps the token head of an ordinary language model and adds two pieces: a predictor that imagines the next state before writing, and an energy that chooses among those futures. Chapter 17 explains each part of the name and what has already been measured for each one.
The name says what the system does: antever, to foresee. An ordinary language model picks the next token and finds out afterwards where it ended up. Ours, before choosing, imagines where each candidate would take it — in latent space, without decoding — and chooses by the energy of that imagined future. The piece that does this is an action-conditioned world model: given the current state and a candidate token, it predicts the next state. In the code it is called mundo (world); the earlier version, which only predicted the direction without knowing the action, is called compass (compass) and stays in the repository as a control.
Language models predict symbols: a distribution over thousands of tokens, punished by cross-entropy. The JEPA lineage, in computer vision, proposed predicting in latent space — because predicting pixels forces the model to spend capacity on irrelevant detail. And the 2025–2026 proposal (LeJEPA, LeWorldModel) went further: a world model trained end to end, with no EMA teacher, with a Gaussianity regularizer preventing collapse, and the decision made by energy minimization, not by sampling.
Antivision does both: it keeps token prediction and trains the world model on the same states. At generation time, the world model imagines each candidate a few steps ahead, and the choice is made by energy — plausibility plus distance to the heading you wrote. When the plan does not give the predicted result, what gets corrected is the map, not the generator.
Antivision is the structure: the arrangement of parts in this repository — encoder, world model, memory, rudder, token head. Latent Rehearsal is the concept the structure realises: rehearsing in the latent space before speaking. Before emitting a token, the system rolls the candidates a few steps forward in latent space, measures the energy of each rehearsed future, and only then decides. The token stops being the decision and becomes the output.
What is LeCun's and what is ours, stated plainly. The underlying ideas are his: JEPA, predicting in the latent space rather than in data space, collapse and how to avoid it, inference by energy, hierarchy, persistent memory. This collection reads two of his talks — the lecture on world models and the interview — and marks the quotations as his words.
Applying it to text is our part. LeCun takes JEPA to the physical world through video, and treats language as the easier case. Latent Rehearsal tests the same idea on the ground where LLMs already work well: it brings JEPA to text and keeps the autoregressive head, which moves from decision-maker to decoder, guardrail and comparable ruler (chapter 09, §1). It is not a contest between the two paths; it is an engineering question about what happens when they are combined. The chapters measure the answer, one at a time.
An ordinary language model accepts control only through the prompt: you write the beginning and hope. Here there is a second channel, which acts on every token without touching the text — and which can be hierarchical: a level over blocks of tokens gives the short-range target to the token level. The pieces are cheap: each predictor is two
Whether it works or not is another question — and the entire project is organized to answer that with numbers, not impressions. Section 3 brings what has already been measured, including what did not work.
| # | Pillar | The limit it starts from | Chapter |
|---|---|---|---|
| 1 | Our own tokenizer | GPT-2's BPE spends 1.6× more tokens on Portuguese and splits numbers into arbitrary pieces | 01 |
| 2 | Geometry as an objective | Training creates a cone in the embedding space: mean cosine from 0.00 to 0.38 | 03, 04 |
| 3 | The world model and the rudder | There is no control channel beyond the prompt, and the model does not know where it will end up before going | 05, 09 |
There is also a fourth axis, which is not a new idea but is mandatory: the modern architecture — RoPE, RMSNorm, SwiGLU, no biases. None of this is our invention; all of it became standard after 2019. What is ours is having both versions in the same file, selectable by a single variable, so the comparison is honest (chapter 02). And a fifth, which cuts across all of them: the method — same seed, same data, always the best checkpoint, every metric with its floor (chapter 06).
All the numbers below came from this machine, with scripts that are in the repository. "Control" is always the same frozen corpus, the same budget of 700 steps and the same seed; the comparison baseline is the run with the cosine regularizer (validation 5.2706; chapter 13 §12.3 explains why the newer tables use 5.2577). Where the measurement contradicted expectation, it is marked.
| Measurement | Result | Reading |
|---|---|---|
| Tokens in Portuguese vs GPT-2 | −47.6% | with a vocabulary 3× smaller; accented text, −73.3% |
| Tokens in code | −12.0% | |
| Tokens in numbers | +58.3% | deliberate worsening — separated digits |
| bits/character in PT ours 7.4 M vs gpt2 124 M | 2.406 vs 3.139 | fewer bits than a model 17× larger — through the tokenizer and the data, not through size |
| Architecture duel, same budget | 192.8 vs 254.5 | PPL: modern against faithful GPT-2 |
| Anisotropy, without → with regularizer | 0.38 → 0.049 | cost: 0.17% of quality |
| Never-seen tokens in the top-50 | 0 | chance would give 1.44 — hypothesis not confirmed |
| SigReg (LeJEPA 2025) in place of cosine | 245.0 vs 244.8 | effective rank — bought nothing, and cost +1.51% |
| VICReg added to cosine | 253.0 +2.88% | the best geometry (decorrelation 0.017) at the highest quality cost |
| DINO: EMA teacher on the deep rudder | +2.08% | the slow teacher cost 3 points against stop-gradient (−0.90%); one seed |
| Barlow Twins on the deep rudder | −0.50% | second run below the control; within the noise of the rudder alone |
| Shallow rudder (predicts the next embedding) | 0.534 floor 0.46 | cosine with the target; real gain +0.07 over copying the state (chance 0.21). Cost +1.41% |
| Deep rudder (predicts the next state) | −0.90% | the only piece that improved the generator: validation 5.2233, PPL 185.55. One seed: caution |
| Additive term α at generation | worse | worse than identity, worse than a random direction — condemned additive inference and motivated the energy |
| World-model correction loop | 0.907 → 0.719 | surprise/floor ratio on a held-out set, smoke test (chapter 09) |
| World model (v3), twelve runs | cos 0.93 vs 0.58 | the map predicts the next state and understands sequence (reversed sentence: error +32%, identity −3%). End to end with weight 1.0 costs the generator +9.59%; stop-grad +1.11%; in two stages on the DINO encoder, zero cost (chapter 05) |
| The rudder governs | 13 of 13 | learned cost head with ten sentences + cost on the real state + γ = 0.1: more heading by the independent judge on eleven of the twelve maps of the queue (chapter 12); the twelfth, the memory map, governs with memory-dropout (chapter 13) |
| Abstraction predicts further | 5 of 5 | level 2 in one step errs less than level 1 rolled eight times, in every hierarchical model without collapse (chapter 11) |
| Persistent memory | +0.019 | the memory that crossed an entire window is worth as much as the recent one (chapter 11) |
| Two collapses caught and corrected | 33× → 1.5× | the ζ that leaked the target (ζ-dropout closed the channel) and the level-2 encoder that shrank 13× (to Gaussianize cost +11.32%; fixing only the scale collapsed the direction — the learned level-2 encoder remains open, the raw mean is the honest abstraction) |
SigReg improved nothing, VICReg cost 2.88%, DINO's EMA teacher cost 2.08%, the shallow rudder costs 1.41% for a gain of 0.07, the additive term makes generation worse, and the end-to-end world model cost the generator up to 9.59%. They are in the main table, with the same prominence as the positive results, because a negative result is as informative as a positive one, and leaving it out would distort the reading of the whole. The shallow-rudder row was already published wrong once, as "0.534 against ~0.00 for chance"; the identity floor is 0.46. Chapter 06 tells that mistake and the other ten.
| # | Chapter | Answers |
|---|---|---|
| 00 | Project map | what we are building and why |
| 01 | Our own tokenizer | why 16k suits this corpus better than 50k, and the price of separated digits |
| 02 | Two architectures, one file | RoPE, RMSNorm, SwiGLU — and the measured duel |
| 03 | The geometry of the embedding | anisotropy, effective rank, decorrelation: the instruments |
| 04 | The lineage of collapse | Barlow Twins → VICReg → JEPA → DINO → SigReg, each link measured on ours: two helped, three cost |
| 05 | The rudder and the world model | the three versions measured; the world-model cells wait for the queue |
| 06 | Method and mistakes | how we measure, and the errors the method caught |
| 07 | Scale and budget | Chinchilla, VRAM, and what US$ 5 buys |
| 08 | The workbench | serving, x-raying and training from the browser |
| 09 | The hierarchical architecture | the design aligned with the world model: energy, MPC, hierarchy, correction |
| 10 | What JEPA is, according to LeCun | the two transcripts read concept by concept, with the file where each one applies |
| 11 | Memory and horizon | persistent memory and "abstraction predicts further": method ready, numbers in the queue |
| 12 | Learned costs and inference modes | cost heads, guardrails, the four modes and the independent judge |
| 13 | The corrections from LeCun | the four failures of the queue taken back to LeCun's text: memory-dropout, VICReg on level 2, EMA target, mixed head |
| 14 | Consulted memory | a bank of abstract states read by attention, persisted on disk by turbovec (4-bit TurboQuant): the memory that survives the session |
| 15 | What it costs and what it delivers | the outward comparison: bits per character against GPT‑1 and GPT‑2, by domain, with each training run priced in dollars and the ceiling the corpus puts on size |
| 16 | The minimum and the volume | the lecture's two claims about energy, measured: whether the search by energy finds the minimum, and how much of the space can get low energy |
| 17 | What an ELLM is | the name of the class of model Antivision builds: each part of energy-based latent predictive language model, with the measurement that supports it |
| 18 | Searching better does not fix the energy | the exact search over the vocabulary and inside the policy's nucleus, measured on the text: all three searches land at the same point; the energy is what decides |
| 19 | The inversion that did not repeat | three seeds on the ladder: the world model costs the generator about 1%, and does not pay 3.38%; the drawn validation swings more than the effect; with fixed batches the cost is +1.77% on mini and +4.92% on pequeno |
| 20 | What changed on 17/09 | update note: the new chapters, the corrections, what changed in the code and what stayed open |
# 1) corpus — Portuguese + English + code, by streaming .venv\Scripts\python.exe model\data\build_corpus.py --mb 1000 # 2) our own tokenizer .venv\Scripts\python.exe model\tokenizer\train.py --vocab 16384 # 3) packs into uint16 tokens (memmap) .venv\Scripts\python.exe model\prepare_data.py # 4) trains — resumes on its own if interrupted with Ctrl+C .venv\Scripts\python.exe model\train.py --size mini --modern --steps 5000 # 4b) the same, with the hierarchical world model .venv\Scripts\python.exe model\train.py --size mini --modern --world 1 --world-levels 2 --sigreg-z 1 # 5) the report: every run, every measurement .venv\Scripts\python.exe model\report.py
Step 4 accepts the flags for each idea in the lineage: --reg-cos, --sigreg, --vicreg, --barlow, --dino, --latent, and those of the world model, --world, --sigreg-z, --world-latent, --world-enc2. All can be switched off, all are comparable with each other, because the name of the output folder carries the configuration.
Think of a chemistry lab with a bench notebook. Every flask carries the label of what was mixed in it — not "experiment 7", but "cosine 1.0 + deep rudder". When someone asks six months later why that flask turned purple, the answer is on the label, not in the memory of whoever mixed it.
That is why mini-moderno-reg1-wm1n2-sz1-duelo is ugly and is right.
Worth saying, because the wrong expectation spoils the reading.
wm* runs) is still in the queue. The central question — whether predicting with the action beats the floor of copying the state, and whether SigReg on the states holds off collapse without an EMA teacher — has no number.Chapter 01 starts where the model starts: turning text into numbers —
and showing why GPT-2's choice costs dearly in any language other than English.