Antivision: an ELLM, a language model that rehearses in latent space before speaking

An LLM picks the next token and finds out afterwards where it ended up. We built one that imagines first: an ELLM. The name says what the system does: antever, to foresee. Before emitting a token, the model rolls each candidate a few steps forward in latent space, without decoding, measures the energy of each imagined future, and only then decides. The token stops being the decision and becomes the output. We call the concept Latent Rehearsal. The underlying ideas are Yann LeCun's. Applying it to text is our part. Seven of the rows in the main table are negative results, and they are there with the same prominence as the positive results.

The three points

1. Three pillars: an encoder that represents, a world model that predicts the next state given the token, and an energy that chooses among imagined futures.
2. Two names, two things: Antivision is the structure. Latent Rehearsal is the concept it realises.
3. What is LeCun's and what is ours, stated plainly: JEPA, energy, anti-collapse and hierarchy are his. Bringing that back to text and keeping the autoregressive head as a decoder is our part, and the chapters measure whether it works.

From the chapter, the beginning

What we are building, why it is not a copy of GPT-2, and what this project's own contribution is — with what has already been measured, including what went wrong.

The previous course took apart models that already existed. This one builds one. The difference is not size — our model is smaller than any of those. The difference is that here every decision was measured, and several follow paths the field found after 2019, on the foundation GPT-2 left.

What it builds has a name: an ELLM, energy-based latent predictive language model — a language model that predicts in latent space and decides by energy. It keeps the token head of an ordinary language model and adds two pieces: a predictor that imagines the next state before writing, and an energy that chooses among those futures. Chapter 17 explains each part of the name and what has already been measured for each one.

The name says what the system does: antever, to foresee. An ordinary language model picks the next token and finds out afterwards where it ended up. Ours, before choosing, imagines where each candidate would take it — in latent space, without decoding — and chooses by the energy of that imagined future. The piece that does this is an action-conditioned world model: given the current state and a candidate token, it predicts the next state. In the code it is called mundo (world); the earlier version, which only predicted the direction without knowing the action, is called compass (compass) and stays in the repository as a control.