The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows

A 7-million-parameter model used fewer bits than the 124-million GPT-2 in Portuguese. Not because it is better. GPT-2's BPE was trained on 40 GB of English. In English, 'tion' appears a lot and became a token. 'ção' did not. A Portuguese text costs 1.6 times more tokens, and every token costs context, quadratic attention and a serial generation step. We trained a 16-thousand-token tokenizer on our corpus. Running Portuguese dropped 47.6%. Accented Portuguese dropped 73.3%. And then our mini, 17 times smaller, came in below GPT-2 in bits per character in Portuguese. It would be easy to announce that and stop. It would be dishonest.

The three points

1. Vocabulary 3 times smaller, embedding table 3 times smaller: 12.9 M parameters became 4.2 M in that piece alone.
2. Split digits cost 58.3% more on text with numbers, and are the choice of Llama, Mistral and Qwen to add better. The benefit in arithmetic has not been measured here yet.
3. The advantage exists only where the tokenizer charges the difference and vanishes where GPT-2 has corpus. In English, it is a tie. What the table proves is that a badly fitted tokenizer costs more than 17× the parameters.

From the chapter, the beginning

Why 16 thousand tokens suit this corpus better than 50 thousand, what we gain by making numbers worse on purpose, and the side effect that was not in the plan.

GPT-2's BPE was trained on WebText: 40 GB of pages in English. The algorithm, as we saw in the previous course, picks the pieces that appear a lot. In English, tion appears a lot. In Portuguese, ção appeared rarely in that corpus — so it never became a token.