Chapter 15 recorded that, with enough data, the world model stopped costing the generator and started paying 3.38%. It was a single seed. With more seeds, and then with a validation measure that stopped swinging, the cost shows up in every round — and it grows with the size of the model.
Chapter 15 trained mini on the whole corpus, with the Chinchilla budget (1,961 steps, 64 million tokens), in two ways: generator only, and generator with the hierarchical world model with EMA target — the recipe that had done best in chapter 13. The generator alone reached 3.7503 validation loss; with the world model, 3.6236. That is 3.38% better, and the chapter read the result as an inversion: in the collection's short training the map cost about 1%, and with enough data it would start paying.
The reading had a reasonable explanation — with little data, predicting the next state competes with the generator for the same capacity; with a lot, it acts as a regulariser — and it matched what one expects from a predictive architecture. But it came from one round on each side, with a single seed, in a regime where the seed spread had never been measured. Chapter 13 §12.3 measured that spread on the duel: about 0.84% of the loss. On the ladder, nobody had measured it.
On 17/09, on an RTX 5090, the same pair was trained with seeds 2 and 3, with the same steps, the same batch and the same corpus. First, the round repeated the duel control, which gave 5.2577 validation, the same number as the 11/09 card to the fourth decimal: the machine is not the source of the difference. That round also brought the measurements for chapters 13 and 16.
Its result pointed at a problem in the ruler itself (§4), so there was a second round the same day, with validation measured another way: six mini rounds and four pequeno rounds, all with --eval-seed, the new option that makes every evaluation read the same batches.
| seed | generator only | + map | difference |
|---|---|---|---|
| 1337 (chapter 15's) | 3.7503 | 3.6236 | −3.38% |
| 2 | 3.5762 | 3.6401 | +1.79% |
| 3 | 3.6091 | 3.7974 | +5.22% |
| mean ± spread | 3.6452 ± 0.0925 | 3.6870 ± 0.0959 | +1.15% |
The data did not confirm the inversion. On both new seeds the map costs, and the mean of the three lands at +1.15% — the same cost chapter 13 had measured in short training. Seed 1337 is the only one where the model with the map comes out below, and it is also the seed where the generator alone had the worst result of the three (3.7503).
But the spread here is too wide to claim anything: 0.0925 for the generator alone, 2.5% of the loss, against the duel's 0.84%. The difference between the means, 0.042, is less than half of that spread. With this ruler, neither the inversion nor the cost holds up.
The large spread does not come only from initialization. It starts inside each round. At every evaluation, training draws 20 new batches from the validation set and measures the loss on them; two consecutive evaluations of the same model barely change the weights, but they change the sample.
| round | step 1147 | 1311 | 1475 | 1639 | 1803 | 1960 |
|---|---|---|---|---|---|---|
| generator only, seed 2 | 3.964 | 3.819 | 3.735 | 3.576 | 3.648 | 3.719 |
| generator only, seed 1337 | 3.919 | 3.878 | 3.750 | 3.764 | 3.799 | 3.759 |
| + map, seed 2 | 4.020 | 3.773 | 3.687 | 3.821 | 3.640 | 3.780 |
On seed 2, the generator alone scores 3.576 at step 1639 and 3.719 at the end of training. That is 0.14 between two measurements of the same nearly finished model — more than the 0.127 that separated the columns on seed 1337 and supported the inversion. The "best checkpoint" is the smallest of 12 numbers like these: in each round it picks the evaluation that landed on the easiest sample. Comparing two minima of noisy measurements largely compares who got luckier in the draw.
The fix is small and additive: model/train.py --eval-seed <n> makes every evaluation read the same validation batches. Without the option, behaviour stays as before, so comparisons with what is already recorded are not broken.
With fixed batches, the whole pair was trained again: three mini seeds and two pequeno seeds, on both sides. The numbers in this section do not compare with those in §3 — they are different validation samples — but they compare with each other, which is what the question asks.
| model | seed | generator only | + map | difference |
|---|---|---|---|---|
mini | 1337 | 3.6729 | 3.6841 | +0.30% |
| 2 | 3.6237 | 3.7141 | +2.50% | |
| 3 | 3.6136 | 3.7051 | +2.53% | |
| mean ± spread | 3.6367 ± 0.0318 | 3.7011 ± 0.0154 | +1.77% | |
pequeno | 1337 | 2.9478 | 3.0978 | +5.09% |
| 2 | 2.9475 | 3.0875 | +4.75% | |
| mean ± spread | 2.9477 ± 0.0002 | 3.0927 ± 0.0073 | +4.92% |
Three things change status with this table.
mini — from 2.5% to 0.87% of the loss, practically the same 0.84% chapter 13 measured on the duel. On pequeno, the two seeds end 0.0003 apart. Drawing the validation accounted for almost all the variation that looked like initialization.mini the mean is +1.77%, twice the seed spread; on pequeno, +4.92%, twenty times the spread.mini to +4.92% on pequeno, with the same loss weight and the same EMA decay. That is the opposite of what the inversion reading suggested — that more budget would make the map pay.| round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
mini, generator only | 5.724 | 5.069 | 4.721 | 4.479 | 4.259 | 4.073 | 3.936 | 3.835 | 3.766 | 3.720 | 3.692 | 3.673 |
mini, + map | 5.778 | 5.117 | 4.762 | 4.495 | 4.227 | 4.015 | 3.888 | 3.807 | 3.756 | 3.723 | 3.699 | 3.684 |
pequeno, generator only | 4.342 | 3.598 | 3.393 | 3.283 | 3.196 | 3.137 | 3.084 | 3.037 | 3.004 | 2.975 | 2.960 | 2.948 |
pequeno, + map | 4.485 | 3.634 | 3.416 | 3.310 | 3.241 | 3.197 | 3.159 | 3.137 | 3.117 | 3.105 | 3.100 | 3.098 |
The curves show where the cost is born. On mini, the model with the map is better in mid-training (3.888 against 3.936 at the seventh evaluation) and only loses at the end, when the generator alone keeps descending and it slows down. On pequeno, the same happens earlier and more strongly: from the sixth evaluation on, the curve with the map flattens, and the gap between the two grows until the end. The bigger the model, the earlier the map's loss seems to compete with the generator's.
Chapter 13 §12.3's rule — a difference below about 0.9% of the loss is not a result — was measured on the duel, with drawn validation, and it still holds for whatever is recorded under that ruler. With fixed validation batches, the seed spread on the ladder is the same 0.87% on mini and almost zero on pequeno, and small differences become legible again. From here on, ladder rounds run with --eval-seed.
| where | before | now |
|---|---|---|
| 15 §5 | "at a real budget the world model stops costing and starts paying": −3.38% | a table with both rulers: the map costs +1.77% on mini and +4.92% on pequeno; the inversion reading was withdrawn |
| 17, table of measurements | "3.38% better (PPL 37.5 against 42.5)" | the map's cost to the generator, with fixed validation, at both sizes |
model/train.py | validation batches drawn at every evaluation | --eval-seed option: fixed batches; without it, the previous behaviour |
The collection's sentences about the world model's cost to the generator, measured on 5.7 million tokens, still hold at 64 and at 212 million. What does not hold is the idea that more data alone turns that cost into a gain: with this recipe, the cost persists and grows with size. What the map delivers in return is measured in other chapters — the heading that steers generation (12), violation of expectation and the guardrail (13) — and none of that shows up in validation loss.
mini and two on pequeno. Three is the minimum to estimate a spread, and the estimated spread itself carries large uncertainty.pequeno trains 3.3 times more steps than mini; separating the two calls for a mini with pequeno's budget.pequeno, where it costs 5%? The battery of pequeno with the map showed level 2 predicting the next block well, but the heading not steering; repeating that with both seeds would say whether it is a property of the size.The sources of the ideas and the method used here. The numbers were measured in the project.
Trained on an RTX 5090 on 17/09/2026 (cloud/round_1709.sh and cloud/round_1709b.sh). Data in out/gpu_1709/, model/runs/*-escala and model/runs/*-escalafix.