What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers

"Our model has perplexity 31, GPT-2 has 29, so GPT-2 is better." That means nothing. Perplexity is measured per token, and a token is not a universal unit: it is a tokenizer's choice. Ours has 16 thousand entries; GPT-2's, 50 thousand. Comparing the two by perplexity is comparing different rulers. The conversion exists and is old: bits per character. How many bits the model needs, on average, to predict the next character of a text it has never seen. With that ruler, we put our models, GPT-1 and GPT-2 in the same table, with the price of each training in dollars. And the expected result was written before seeing the number: ours does better in Portuguese and code, GPT-2 does better in English. If ours does better in English too, suspicion falls on the test, not the model.

The three points

1. The parameter saving is in the vocabulary: our biggest model has the same body as GPT-2 small (12 layers, 768 dimensions), but GPT-2's embedding is 31% of the total, and in our mini it is 56.6% of a model 17 times smaller.
2. The card changed the scale of the possible: the 700-step run that takes 25 minutes on CPU takes 40 seconds on an RTX 5090. Training the mini with the Chinchilla budget costs US$ 0.03; the large, US$ 4.28 and 4.8 hours.
3. The rule was written before the number: the test text comes from the validation set, the domain of each block is decided by code, and a result that favours whoever measures deserves more suspicion, not less.

From the chapter, the beginning

Every earlier chapter compares our own rounds against each other. This one compares outward: GPT‑1, GPT‑2 and ours, on the only metric that is valid across different tokenizers, with the dollar price of each training run. The question is simple and uncomfortable: a model with fewer parameters, trained for less than a dollar, delivers what exactly?

The temptation is to say "our model has perplexity 31 and GPT‑2 has 29, so GPT‑2 is better". That means nothing. Perplexity is measured per token, and a token is not a universal unit: it is a choice made by the tokenizer. Our vocabulary has 16,392 entries trained on this corpus; GPT‑2's has 50,257, trained on English web text. The word "antevisão" may become two pieces in ours and six in GPT‑2's. Per-token perplexity compares two things counting different units — like comparing litres per kilometre with gallons per mile without converting.

The conversion exists and is old: bits per character. Sum the negative log-likelihood of every token in a text, convert from nats to bits, and divide by the number of characters — not tokens. The result answers a physical question: how many bits does this model need, on average, to transmit one character of this text? A character is the same thing for everybody. It is the metric of the compression literature, and the only honest one here.

No hand-picked text. The test text comes from the validation set itself (model/data/validacao.bin), decoded back into characters: blocks of 8,192 tokens that data preparation set aside before training and shuffled. No model of ours has seen them. Each block's domain — Portuguese, English or code — is decided by a counting rule written into the code (model/cost_vs_gpt2.py), not by judgement: structural code markers, Portuguese markers, and the rest is English.