Searching better does not fix the energy: three searches, the same text

I fixed my model's search so it finds the true lowest energy, among 16,392 options. The text did not get better. The previous measurement had shown that my generator looked at only 8 candidates and missed the energy minimum. So I wrote the exact search. At first it gave worse text — and the reason was a defect of mine: the search ignored the term that measures whether a word is plausible. Once fixed, I tested three places to search: the 8 most likely tokens, the whole vocabulary, and the nucleus of the distribution. With the same plausibility weight, both exact searches landed at the same point: topic affinity 0.48, perplexity 83. Exactly the point the short list already reached by lowering that weight. Searching better moves along the trade-off between following the topic and writing something plausible. It does not shift it.

The three points

1. The exact search's first result was a defect: it ordered the vocabulary by the heading alone, without the plausibility term, and perplexity reached 2,098. With the whole energy, it fell to 83.
2. Three searches — the 8 most likely with a lower plausibility weight, the whole vocabulary, and the policy's nucleus — land at the same point: affinity between 0.476 and 0.488, perplexity between 82 and 84.
3. What holds plausibility is the token's rank in the policy's list, not probability mass: the model's policy is so flat that the 80% nucleus holds 2,000 tokens, and the tokens the energy picks sit between rank 9 and rank 128.

From the chapter, the beginning

Chapter 16 showed that the energy search did not find the minimum and built an exact search. This chapter measures what the exact search does to the text, and the result points somewhere else: the trade-off between following the heading and writing something plausible is set by the energy, not by where one searches.

In energy mode, the generator picks each token in two steps. First it builds a short list of candidates. Then it imagines, in latent space, where each candidate would take the text, and gives each one an energy with two terms: how far the predicted state lies from the requested heading, and how improbable the token is according to the token head itself.

Chapter 16 showed that the short list — the 8 most probable tokens — often does not contain the lowest-energy token: at γ = 0 this happens in all 96 positions measured. The response was a new mode, todos (all), which computes the energy of each of the 16,392 tokens in the vocabulary and takes the lowest. One question remained, the first on that chapter's list of open questions: does finding the true minimum improve the text, or only lower the energy?