The inversion that did not repeat: the world model's cost appears once the ruler stops swinging

I wrote in a chapter that my world model made the generator 3.38% better. With more seeds and a fixed ruler, it costs. The result came from a single round: the generator alone at 3.7503 loss, with the world model at 3.6236. With seeds 2 and 3, the map went back to costing. And while investigating why, I found the problem in the ruler: at every evaluation training drew a new validation sample, and the same nearly finished model scored 3.576 at one step and 3.719 at the end — a swing larger than the effect I was measuring. I fixed the validation batches and redid the whole pair: three mini seeds and two pequeno seeds. With the clean ruler, the seed spread fell from 2.5% to 0.87% of the loss, and the world model's cost showed up in every round: +1.77% on mini, +4.92% on pequeno.

The three points

1. With seeds 2 and 3, the world model goes back to costing. The −3.38% came from a favourable seed, under a protocol where the drawn validation swung more than the effect itself: 3.576 at one step, 3.719 at the end of the same run.
2. With fixed validation batches, the seed spread falls from 2.5% to 0.87% of the loss on mini, and to almost zero on pequeno. Nearly all the noise came from drawing the sample, not from initialization.
3. With the clean ruler, the cost shows up in all five comparisons: +1.77% on mini (3 seeds) and +4.92% on pequeno (2 seeds). More budget does not flip the sign; the cost grows with the size of the model.

From the chapter, the beginning

Chapter 15 recorded that, with enough data, the world model stopped costing the generator and started paying 3.38%. It was a single seed. With more seeds, and then with a validation measure that stopped swinging, the cost shows up in every round — and it grows with the size of the model.

Chapter 15 trained mini on the whole corpus, with the Chinchilla budget (1,961 steps, 64 million tokens), in two ways: generator only, and generator with the hierarchical world model with EMA target — the recipe that had done best in chapter 13. The generator alone reached 3.7503 validation loss; with the world model, 3.6236. That is 3.38% better, and the chapter read the result as an inversion: in the collection's short training the map cost about 1%, and with enough data it would start paying.

The reading had a reasonable explanation — with little data, predicting the next state competes with the generator for the same capacity; with a lot, it acts as a regulariser — and it matched what one expects from a predictive architecture. But it came from one round on each side, with a single seed, in a regime where the seed spread had never been measured. Chapter 13 §12.3 measured that spread on the duel: about 0.84% of the loss. On the ladder, nobody had measured it.