Every earlier chapter compares our own rounds against each other. This one compares outward: GPT‑1, GPT‑2 and ours, on the only metric that is valid across different tokenizers, with the dollar price of each training run. The question is simple and uncomfortable: a model with fewer parameters, trained for less than a dollar, delivers what exactly?
The temptation is to say "our model has perplexity 31 and GPT‑2 has 29, so GPT‑2 is better". That means nothing. Perplexity is measured per token, and a token is not a universal unit: it is a choice made by the tokenizer. Our vocabulary has 16,392 entries trained on this corpus; GPT‑2's has 50,257, trained on English web text. The word "antevisão" may become two pieces in ours and six in GPT‑2's. Per-token perplexity compares two things counting different units — like comparing litres per kilometre with gallons per mile without converting.
The conversion exists and is old: bits per character. Sum the negative log-likelihood of every token in a text, convert from nats to bits, and divide by the number of characters — not tokens. The result answers a physical question: how many bits does this model need, on average, to transmit one character of this text? A character is the same thing for everybody. It is the metric of the compression literature, and the only honest one here.
No hand-picked text. The test text comes from the validation set itself (model/data/validacao.bin), decoded back into characters: blocks of 8,192 tokens that data preparation set aside before training and shuffled. No model of ours has seen them. Each block's domain — Portuguese, English or code — is decided by a counting rule written into the code (model/cost_vs_gpt2.py), not by judgement: structural code markers, Portuguese markers, and the rest is English.
Our largest model has the same body geometry as GPT‑2 small: 12 layers, 12 heads, 768 dimensions. The entire difference is the vocabulary, and it is not small.
| model | total | embedding | body | embedding as % of total | layers × d |
|---|---|---|---|---|---|
mini | 7,410,432 | 4,196,608 | 3,213,824 | 56.6% | 4 × 256 |
pequeno | 16,917,120 | 6,294,912 | 10,622,208 | 37.2% | 6 × 384 |
medio | 34,092,544 | 8,393,216 | 25,699,328 | 24.6% | 8 × 512 |
grande | 97,544,448 | 12,589,824 | 84,954,624 | 12.9% | 12 × 768 |
| GPT‑2 small | 124,439,808 | 38,597,376 | 85,054,464 | 31.0% | 12 × 768 |
| GPT‑1 | 116,534,784 | 31,087,104 | 85,054,464 | 26.7% | 12 × 768 |
The body is the same; the vocabulary is the weight. Our grande has 97.5 million parameters against GPT‑2 small's 124.4 — 21.6% fewer — and both have essentially the same body, 85 million. The 26.9 million of difference is almost entirely translation table: 50,257 entries against 16,392. That is not cleverness on our part, it is the consequence of training the tokenizer on this corpus instead of inheriting one. Chapter 01 measured the other side of the same coin: a vocabulary fitted to the corpus spends fewer tokens per character, so the same context window carries more text.
Every round so far was measured on CPU, and that is a choice of method: it is what lets anyone repeat the whole experiment without depending on a card. The price of that choice is time — 700 steps take 25 minutes. The same 700 steps on an RTX 5090 take 40 seconds. The difference is not comfort: it is what makes it possible to measure the whole ladder of sizes in one sitting, instead of over a weekend.
| model | CPU (tok/s) | RTX 5090 (tok/s) | how many times faster |
|---|---|---|---|
mini | ≈ 3,800 | 575,752 | 151× |
pequeno | 2,185 | 509,073 | 233× |
medio | 1,203 | 225,341 | 187× |
grande | 361 | 98,725 | 273× |
The reading that matters is not "GPUs are fast", which everyone knows. It is the shape of the curve: from mini to pequeno the card barely loses speed (576k to 509k tokens per second) while the model's body triples. At those sizes the card is idle — the bottleneck is the small batch, not the arithmetic. Only from medio on does the curve behave as expected, falling with size. Training small models on a big card means paying for silicon that sits still, and the table below shows how much.
The CPU has no such idle regime: it slows with size from the first step (4,108, 2,185, 1,203, 361 tok/s). That is why the card's advantage grows with the model, from 151× on mini to 273× on grande. On this CPU, the 1.7 billion tokens Chinchilla asks for grande would take about 54 days; on the card, 4.8 hours.
Chinchilla's rule (chapter 07) says efficient training uses about 20 tokens per parameter. Chapter 07 applied it to the body and not to the whole model, for the same reason as section 2: the embedding is a table, not compute capacity. Applied to our 309,083,117 training tokens:
| model | body | budget (20× body) | corpus epochs | time on the 5090 | cost |
|---|---|---|---|---|---|
mini | 3.2 M | 64 M tokens | 0.21 | 1.9 min | US$ 0.03 |
pequeno | 10.6 M | 212 M tokens | 0.69 | 7.0 min | US$ 0.10 |
medio | 25.7 M | 514 M tokens | 1.66 | 38 min | US$ 0.57 |
grande | 85.0 M | 1.70 G tokens | 5.50 | 4.8 h | US$ 4.28 |
We do not have the corpus for the large model, and that is the result, not an obstacle to hide. Training 85 million body parameters to Chinchilla's optimum would ask for 1.7 billion tokens; the corpus has 309 million. That would be five and a half epochs over the same text. Past that, the model memorises instead of generalising, and validation loss stops following training loss. medio already asks for 1.66 epochs — it sits at the edge of what the corpus sustains. This collection's practical conclusion is recorded here: from about 34 million parameters on, our limit stopped being the card and became the corpus. More GPU buys no more quality; more text does.
The first two rungs, trained on the full corpus with section 4's budget. The cells fill when each round closes.
| model | steps | tokens seen | validation loss | PPL | time | cost |
|---|---|---|---|---|---|---|
duel control (mini, 60 M tokens) | 700 | 5.7 M | 5.2577 | 192.0 | 40 s | US$ 0.010 |
mini | 1,961 | 64 M | 3.7503 | 42.5 | 1 min 25 s | US$ 0.021 |
pequeno | 6,483 | 212 M | 2.8611 | 17.5 | 7 min 06 s | US$ 0.106 |
The budget matters more than the size. The same mini, same architecture and same 7.4 million parameters, goes from perplexity 192 to 42.5 — four and a half times better — purely by seeing 64 million tokens instead of the duel round's 5.7 million. Not one line of the model changed: what changed is how much it read. This collection's whole queue, chapters 03 to 14, ran on 5.7 million tokens per round, because that is what the CPU allowed in twenty minutes. That does not invalidate the comparisons between rounds, which remain fair against each other, but it places all of them in a short-training regime that the card has just shown to be well below what the corpus allows.
The next rung confirms the direction: pequeno, with 2.3× the parameters and 3.3× the tokens, reaches perplexity 17.5 for ten cents. The two runs together cost US$ 0.13 and took eight and a half minutes. That is the number that sets the scale for the rest of this chapter: if eight minutes and thirteen cents buy a jump from 192 to 17.5, what limits us is no longer the money or the card.
| budget | generator only | + world model (EMA) | what the map costs |
|---|---|---|---|
duel, mini — 5.7 M tokens, drawn validation | 5.2744 ± 0.0443 | 5.3351 ± 0.0455 | +1.15% |
ladder, mini — 64 M tokens, seed 1337 only | 3.7503 | 3.6236 | −3.38% |
ladder, mini — 64 M tokens, three seeds, drawn validation | 3.6452 ± 0.0925 | 3.6870 ± 0.0959 | +1.15% |
ladder, mini — three seeds, fixed validation | 3.6367 ± 0.0318 | 3.7011 ± 0.0154 | +1.77% |
ladder, pequeno — 212 M tokens, two seeds, fixed validation | 2.9477 ± 0.0002 | 3.0927 ± 0.0073 | +4.92% |
The first version of this section read the table's second row as an inversion: with eleven times more tokens, the world model would stop costing and start paying 3.38%. The data did not confirm that reading. On seeds 2 and 3 the map went back to costing (+1.79% and +5.22%), and the −3.38% was a favourable seed, under a protocol where the drawn validation swung more than the effect itself.
With fixed validation, which chapter 19 explains, the ruler became clean: the seed spread fell to 0.87% of the loss on mini and to almost zero on pequeno. Under that ruler, the world model costs on every seed of both sizes: +1.77% on mini and +4.92% on pequeno. The cost does not vanish with more budget; it grows with the size of the model.
The world model with hierarchy and EMA target costs the generator at every budget measured: about 1% on the duel, 1.77% on the ladder's mini and 4.92% on pequeno. The collection's sentences about the map's cost, measured on 5.7 million tokens, still hold at 64 and 212 million. What the map delivers in return — the heading, violation of expectation, the guardrail — is in chapters 12 and 13, and does not show up in validation loss. Chapter 19 tells how a single-seed reading ended up in a chapter, and what it changed in the method: training gained --eval-seed, and ladder rounds run with fixed validation.
This chapter's central table. Lower is better. The three domain columns exist because the average hides what matters: a model can do well on Portuguese and badly on English, and reporting only the average would be picking the number that flatters.
| model | parameters | training cost | bits/char (mean) | Portuguese | English | code |
|---|---|---|---|---|---|---|
pequeno (ours) | 16,917,120 | US$ 0.106 | 1.273 | 1.360 | 1.486 | 0.972 |
mini (ours) | 7,410,432 | US$ 0.021 | 1.637 | 1.737 | 1.821 | 1.354 |
| GPT‑2 small | 124,439,808 | — | 1.710 | 2.628 | 1.105 | 1.398 |
| GPT‑1 | 116,534,784 | — | 2.152 | 3.055 | 1.396 | 2.006 |
The expected result is not "ours does better". It is ours does better on Portuguese and code, GPT‑2 does better on English — for a reason with nothing mysterious about it: GPT‑2 was trained on English web text and never saw structured Portuguese, while 39% of our corpus is Portuguese Wikipedia and 33% is Python code. If ours also does better on English, the likeliest explanation is not that our model is better, but that the English test text is too easy or too short — and the table must be redone with more blocks before any celebration. A result that flatters whoever measured it deserves more suspicion than one that contradicts them.
| model | bits/char on plain English | reading |
|---|---|---|
| GPT‑2 small | 1.121 | within expectation |
| GPT‑1 | 1.327 | within expectation |
pequeno (ours) | 1.449 | within expectation |
mini (ours) | 1.564 | at the upper edge |
The first version of this table gave GPT‑2 4.772 bits per character — four times the real value — and our pequeno 1.273. It was too good, and this section's "how to read" box, written before any number existed, said to distrust exactly that. The error was in the measurement, not the model: models from the transformers library shift the labels themselves, and the script was passing labels already shifted. GPT‑2 was being asked to predict two tokens ahead. The sanity row now runs always, and any value above 2.5 is flagged as suspicious in the program's own output.
The result is what the box predicted, with no favourable surprise. Ours does better on Portuguese by a wide margin — 1.360 against GPT‑2's 2.628, nearly double — and on code, 0.972 against 1.398. GPT‑2 does better on English, 1.105 against 1.486, and the explanation holds no mystery: it was trained on English web text, and only 14% of our corpus is English web. Each model does best on what it read.
The number that matters for this collection's thesis is in the first row. pequeno, with 16.9 million parameters against 124.4 — seven and a half times fewer — and ten cents of training, lands at 1.273 bits per character against GPT‑2's 1.710: 26% better across the three domains. And mini, with 7.4 million parameters and two cents, already passes GPT‑2 on the mean (1.637 against 1.710), losing only on English. This does not say we trained better than OpenAI did in 2019; it says a small model trained on its own domain outperforms, in that domain, a model seventeen times larger that was trained for another. It is an argument about data, not about cleverness, it says nothing against GPT‑2 on the English it was built for, and it is section 4's argument seen from the other side.
GPT‑1 trails on everything, including the English where it should be at home (1.396 against GPT‑2's 1.105). Both have the same 85-million-parameter body; what separates them is how much text was read and how big the vocabulary is. It is our ladder's lesson, measured one generation apart.
Bits per character measures one thing: predicting the next piece of text. It is the task GPT‑2 was built for, and comparing only on it is comparing on its home ground. The pieces this collection built do not show up in that metric, and that is precisely why they exist.
| function | what it does | cost to the generator | where measured |
|---|---|---|---|
| world model | predicts the next latent state, not the next token: gives the planner a map to imagine with before writing | +0.45% (EMA target) | 13 §4 |
| energy rudder | picks the token that moves closest to a declared heading, searching in latent space | 0 (inference only) | 12, 13 §4 |
| guardrail on the imagined state | blocks what the model is about to write, not what it already wrote: 0.0% digits on 4 of 4 maps | 0 (inference only) | 13 §5 |
| dependency energy | detects order violation without being given the token: +130% to +142% against +38% for plain error | 0 (inference only) | 13 §11 |
| consulted memory | a bank of abstract states read by attention, persisted to disk with a quantised index | +9.73% (indistinguishable from the map without it) | 14 |
| calibrated surprise | measures when the map was wrong, and hands control back to the policy on those steps | 0 (inference only) | 13 §11 |
The sum is this collection's argument: the functions cost 0.45% of loss and no extra parameters on the generation path, because almost all of them live in inference. These functions need a world model. One can be added to an already trained model such as GPT‑2, but it costs more — chapter 13 measured that a second-stage map over a frozen encoder comes out more expensive than training it jointly.
grande does not fit the corpus. Either the corpus grows to 1.7 billion tokens, or medio is the largest honest model in this collection. Measuring where validation loss detaches from training loss, by size, would say which.