Energy-based latent predictive language model. The right name for what Antivision builds: a language model that predicts in latent space and decides by energy. This chapter explains each word of the name, shows what has already been measured for each one, and what the name does not promise.
Until now this collection has called the system a "language model", and at times compared it with "an LLM". Both miss what it does differently.
The first L in LLM stands for large: the name describes scale. And what an LLM does, by construction, is choose the next token by the probability a forward pass assigns to each candidate. The model in this collection is not large — mini has 7.4 million parameters — and what sets it apart is not size but where it predicts and how it decides.
An ELLM — Energy-based Latent predictive Language Model — is a language model that, before emitting a token, predicts in latent space where each candidate would take the state, and chooses by minimizing an energy over those predicted futures. The token head is still there: it is what turns the state into text.
Latent Rehearsal (rehearsing in the latent space before speaking) is still the name of the idea, and Antivision is still the name of the structure. ELLM is the name of the class of model that structure belongs to. The underlying ideas — energy-based models, predicting in the latent space, JEPA — are Yann LeCun's; the name and the application to text are this project's part.
| part of the name | what it means | the project piece | where it was measured |
|---|---|---|---|
| Energy-based | the token choice comes from minimizing an energy — heading, plausibility and guardrails added together — not from probability alone | the rudder and the inference modes (energy, gradient, mpc) | 12, 16 |
| Latent | prediction happens in the space of internal states, not in the space of text: nothing is decoded while the model imagines | the encoder's hidden states, at levels 1 and 2 | 09, 13 |
| predictive | there is a piece that predicts the next state given the current state and a candidate token — the world model | the action-conditioned mundo predictor | 05, 13, 15 |
| Language Model | the output is still text, token by token, and the model is still trained to predict the next token | the token head, as decoder, guardrail and comparable ruler | 02, 15 |
The comparison below describes each one's mechanism, not a contest. An ELLM contains the central piece of an LLM — the head that predicts the next token — and adds other pieces around it.
| question | LLM | ELLM (this project) |
|---|---|---|
| what it predicts | the next-token distribution | the next-token distribution and the next latent state, for each candidate |
| how it chooses | samples or takes the most likely, in one forward pass | imagines each candidate's future and chooses the lowest energy |
| how it is steered | through the prompt and fine-tuning | through the prompt, and also through a heading and guardrails that enter the energy at every step |
| collapse risk | does not apply to the next-token objective | exists in the world model, and is controlled by regularization (16) |
| how it is measured | perplexity, bits per character | the same, plus energy, search coverage and volume of low energy |
The name only holds if every part of it has a number. These are the results that support the definition, all from earlier chapters:
| part | what was measured | result | chapter |
|---|---|---|---|
| Energy-based | the best rudder configuration (learned head, cost on the real state, γ = 0.1) against free generation | affinity 0.355 against 0.256, with PPL 36.7 against 48.5 | 12 |
| Energy-based | the guardrail in the energy, applied only to the imagined states (mixed head, γc = 3), on a prompt that pulls towards numbers | 0.0% digits, against 50–56% without the guardrail | 13 §5 |
| Latent + predictive | the world model's cost to the generator on the full corpus, fixed validation | +1.77% on mini (3 seeds) and +4.92% on pequeno (2 seeds) | 15 §5 and 19 |
| Latent + predictive | dependency energy detects violated order without being given the token | +130% to +142%, against +38% for the plain error | 13 §11 |
| Latent + predictive | the volume of space with low energy, regularized against shortcut | median energy of a sampled point about 4× cheaper in the shortcut (ratio 3.6–4.1) | 16 §8 and §10 |
| Language Model | the size of the whole world model inside the model | 7% of the parameters | 00 |
Chapter 16 measured an important limit of the energy-based part. With the default guardrail weight (γ = 1), the energy is dominated by the token head's own probability, and the energy minimum lies among the 8 most likely tokens in 100% of positions — and is the most likely token itself in 90.6% of them. The decision only truly belongs to the energy when the heading has weight (low γ). And chapter 18 measured the other side: at low γ, searching better — over the whole vocabulary or inside the policy's nucleus — does not improve the text; it lands at the same affinity and perplexity as the short list with a lower γ. An ELLM, then, is a model that can decide by energy; how much it does depends on the configuration, and that is measured, not assumed.
mini, with 7.4 million parameters.This chapter brings no new run: it names what the earlier ones measured, and points to where each number lives.