ELLM: the right name for a language model that predicts in latent space and decides by energy

I called my model an LLM for months. The name was wrong — and not because of its size. The L in LLM stands for large, and what an LLM does is choose the next token by the probability of a forward pass. My model has 7.4 million parameters and does something else before writing: it imagines, in latent space, where each candidate would take the text, and chooses the lowest energy. The right name for that is ELLM: energy-based latent predictive language model. Every part of the name has a measured number. And one of them has a limit that is also measured: at the default setting, the lowest-energy token is the most likely token itself in 90.6% of positions.

The three points

1. The first L in LLM stands for large, and describes scale. An ELLM is defined by something else: it predicts the next state in latent space for each candidate and chooses the lowest energy. The token head is still there, writing the text.
2. Every part of the name has a measurement: the energy rudder takes affinity from 0.256 to 0.355 with lower perplexity; the world model, with enough data and three seeds, costs the generator about 1%, within the noise; in the regularized model, the energy of an arbitrary point in space is about four times more expensive than in the shortcut.
3. The limit is measured too: at the default guardrail weight, the lowest-energy token is the most likely one in 90.6% of positions. An ELLM can decide by energy; how much it does depends on the configuration.

From the chapter, the beginning

Energy-based latent predictive language model. The right name for what Antivision builds: a language model that predicts in latent space and decides by energy. This chapter explains each word of the name, shows what has already been measured for each one, and what the name does not promise.

Until now this collection has called the system a "language model", and at times compared it with "an LLM". Both miss what it does differently.

The first L in LLM stands for large: the name describes scale. And what an LLM does, by construction, is choose the next token by the probability a forward pass assigns to each candidate. The model in this collection is not large — mini has 7.4 million parameters — and what sets it apart is not size but where it predicts and how it decides.

An ELLM — Energy-based Latent predictive Language Model — is a language model that, before emitting a token, predicts in latent space where each candidate would take the state, and chooses by minimizing an energy over those predicted futures. The token head is still there: it is what turns the state into text.