Why 16 thousand tokens suit this corpus better than 50 thousand, what we gain by making numbers worse on purpose, and the side effect that was not in the plan.
GPT-2's BPE was trained on WebText: 40 GB of pages in English. The algorithm, as we saw in the previous course, picks the pieces that appear a lot. In English, tion appears a lot. In Portuguese, ção appeared rarely in that corpus — so it never became a token.
The consequence is surgical. Look at the most ordinary word in technical Portuguese, 'ação' (action):
'ação' nosso 1: ['ação']
gpt2 3: ['a', 'ç', 'ão']
'implementação' nosso 3: ['im', 'plement', 'ação']
gpt2 5: ['im', 'plement', 'a', 'ç', 'ão']
Note the detail: GPT-2 knows plement. It learned the Latin root through the English implementation. But when it comes to the suffix, it breaks off the ç on its own — because the cedilla, to it, is a strange character that appears in strange texts.
Every token costs three things: a position in the context (which is finite), a pass through attention (whose cost is quadratic in the number of positions), and a generation step (which is serial — 93.3% of the time of a request, as we measured in chapter 08).
A Portuguese text that costs 1.6× more tokens costs 1.6× more context, ~2.5× more attention and 1.6× more time to generate. The model does not just get more expensive: it is left with less useful memory, because the same subject takes up more of the window.
We trained our own BPE on our corpus — Portuguese, English and code. Four choices separate ours from the original, and each has a cost.
Counterintuitive: shouldn't a smaller vocabulary mean longer texts? It does, all else being equal. But all else is not equal, because our corpus is the right corpus. A 16k vocabulary fitted to your text works better than a 50k one fitted to some other text.
And there is a second gain, which only shows up when you count the parameters. The embedding table has
| Vocabulary | Table | Blocks | Total | % in dictionary |
|---|---|---|---|---|
| 50,257 (GPT-2) | 12.9 M | 3.2 M | 16.1 M | 80.1% |
| 16,392 (ours) | 4.2 M | 3.2 M | 7.4 M | 56.6% |
With GPT-2's vocabulary, 80% of our model would be dictionary. The part that learns is the "blocks" column. Shrinking the vocabulary does not just save memory — it frees budget for the layers, which is where reasoning lives.
This is the only change that worsens a metric on purpose. GPT-2 treats numbers like any other text: the pieces come out of frequency.
' 1234567' nosso 8: [' ', '1', '2', '3', '4', '5', '6', '7']
gpt2 3: [' 123', '45', '67']
Three tokens against eight. GPT-2 looks more efficient — and it is, in bytes. But look at what it did: it broke 1 234 567 into 123, 45, 67. Those pieces have nothing to do with the structure of the number. The 45 there is the hundred-thousands and the ten-thousands; in another number, the same token 45 will be the units.
To add two numbers, you have to line up the places. If 45 means one thing in one number and something else in another, the model has to learn arithmetic separately for every BPE cut pattern — and the cuts change from number to number.
With separated digits, every position is explicit and always the same. It is the choice of Llama, Mistral and Qwen, and it helps explain why they do arithmetic better than GPT-2 at comparable sizes.
We pay +58.3% tokens on text with numbers for that consistency. It is a real cost, and chapter 06 explains why we accepted it without having measured the benefit: because measuring it would require training until the model can add, which is far above our current budget. It is a bet grounded in the literature, not a result of ours.
Python code is indentation. We added explicit tokens for 4, 8, 12 and 16 spaces:
' return' nosso 2: [' ', 'return']
gpt2 4: [' ', ' ', ' ', ' return']
GPT-2 spends one token per space. In a Python file with deep nesting, that is half the file. Worse: the model has to count spaces to know the indentation level, which is exactly the kind of task attention is bad at.
tok.pre_tokenizer = pre_tokenizers.Sequence([ pre_tokenizers.Split(pattern=Regex(r"'s|'t|'re|…| ?\p{L}+| ?\p{N}+|…"), behavior="isolated"), pre_tokenizers.Digits(individual_digits=True), # decision 2 pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False), ])
The order matters. Split separates words from punctuation; Digits breaks numbers before BPE can merge them; ByteLevel guarantees that any byte is representable — no <UNK>, ever.
| Sample | Ours | GPT-2 | Difference |
|---|---|---|---|
| Everyday Portuguese | 11 | 21 | −47.6% |
| Portuguese with accents | 4 | 15 | −73.3% |
| Python code | 22 | 25 | −12.0% |
| English | 9 | 9 | 0.0% |
| With numbers | 19 | 12 | +58.3% |
The accents row is the most revealing: "ação coração informação atenção" (action, heart, information, attention) costs 4 tokens for us and 15 for GPT-2. Almost four times. And English ties — with a vocabulary three times smaller, which is already a result in itself.
Tokens are a means, not an end. What matters is whether the model predicts better. The honest metric across different tokenizers is bits per character — perplexity is measured per token, and comparing PPL across different vocabularies means nothing.
| Model | Portuguese | English | Code |
|---|---|---|---|
| our mini (7.4 M) | 2.406 | 2.038 | 2.405 |
| gpt2 (124 M) | 3.139 | 1.261 | 1.864 |
In Portuguese, 7.4 million parameters use fewer bits than 124 million — 23% fewer bits, with 17× less model. In English and code GPT-2 does much better, and that makes sense: it saw ~10 billion tokens against our 5.7 million.
It would be easy to announce "our 7 M model beats GPT-2" and stop there. It would be dishonest. The advantage exists only where the tokenizer charges the difference, and vanishes where GPT-2 has corpus. What the table proves is not that we are better — it is that a badly fitted tokenization costs more than 17× the parameters.
When measuring training speed, something we did not expect showed up: the model with the 16k vocabulary trains at 4,000 tokens/s, against 857 tok/s in the earlier measurement with 50k. Five times faster.
The cause is in the last layer. The projection to the logits is a
It is like a multiple-choice exam. Answering a question with 50 thousand alternatives is not 3× slower than one with 16 thousand because of the reasoning — the reasoning is the same. It is slower because you have to read all the alternatives before choosing. The model does that for every token it generates.
Practical consequence: the mini model went from "days of CPU" to ~4 hours. A decision made for tokenization quality turned out to be, also, the biggest speed optimization in the project. We did not predict this — and that is why we measure instead of estimating.
With text turning into numbers the right way, chapter 02 assembles the model —
in both versions, the 2019 one and today's, in the same file so the difference is legible.