Tokenization
and the BPE algorithm

1 Intuition: the model does not read what you write

A neural network is, at bottom, a matrix-multiplying machine. Matrices are made of numbers. So the first thing that must happen to your text is a translation: letters become numbers. This step is called tokenization, and it happens before the model — it is not learned along with it; it is a fixed table, decided once and frozen forever.

The question is: numbers for what? There are three historical answers, and understanding why the first two failed is the only way to understand why the third won.

Attempt 1 — one number per word

Simple: build a dictionary. "casa"→41, "cachorro"→892, and so on. It works until the day a word shows up that is not in the dictionary — a proper name, a typo, a new slang word, a word in another language. Then the model writes <UNK> ("unknown") and loses the information forever. And the dictionary of real English, with names and variants, goes past a million entries — which would make the last layer of the model (which needs one neuron per possible word) bigger on its own than the entire model.

Attempt 2 — one number per character

This solves the unknown: there are only ~100 characters, and any text fits. But it creates a worse problem. The word "internacionalização" becomes 19 positions instead of 1. Since the cost of attention grows with the square of the number of positions (you will see this in chapter 03), texts become much more expensive. And the model has to spend several layers just to rediscover that c-a-s-a put together means something.

Attempt 3 — one number per frequent piece

Here is the insight. What if the vocabulary were made of pieces of words, chosen automatically by the statistics of the text? Common words ("the", "casa") get a single token. Rare words decompose into pieces that exist ("Rafhael" → R + af + ha + el). Nothing is ever unknown, and the text stays short.

That is BPE — Byte Pair Encoding. It is what GPT-2 uses, it is what RoBERTa uses, and variants of it are still used in many current models.

2 How BPE builds the vocabulary

BPE is a tokenizer training algorithm. It runs once, over a giant corpus, and produces two things: a list of tokens (vocab.json) and an ordered list of merges (merges.txt). After that, it never changes again.

The procedure

1. Start with the alphabet. Each isolated symbol is a token. In GPT-2, that alphabet is the 256 possible bytes (we will come back to this in section 3).
2. Count all neighboring pairs across the whole corpus. How many times does t appear followed by h? And h followed by e?
3. Merge the most frequent pair into a new token. If t+h is the champion, the token th is created, and every t h in the corpus becomes th.
4. Repeat — now th+e may become the champion, creating the.
5. Stop when the vocabulary reaches the desired size. In GPT-2: 50,257 tokens.

A tiny example, done by hand

Corpus: the word banana repeated 5 times, and bandana 2 times. Let's follow three merges.

3 GPT-2's move: BPE at the byte level

Here is GPT-2's specific contribution, and it is elegant. The original BPE (used in machine translation, 2016) started from Unicode characters. Problem: there are more than 149 thousand Unicode characters. If the initial alphabet is already huge, little room is left in the vocabulary for the useful merges — and emojis or Chinese ideograms could still fall out and become <UNK>.

GPT-2's solution: don't start from characters, start from bytes. All digital text, in any language, is a sequence of UTF-8 bytes. And bytes only come in 256 possible values. So:

• The initial alphabet has exactly 256 symbols — small and closed.
• Nothing is ever unknown. Emoji, Chinese, Arabic, a binary file pasted by mistake — everything is bytes, everything is representable. GPT-2 has no <UNK> token. It doesn't need one.
• The cost: an emoji takes 4 UTF-8 bytes, so it may cost 3–4 tokens. Acceptable.

The odd detail: why "Ġ" and "Ċ" show up

When you inspect GPT-2's tokens, you will see things like Ġthe and ĠcasaĊ. It looks strange, but it is a simple convention.

Some bytes are control characters or whitespace — printing them directly would break the vocabulary files. So GPT-2 remaps these problematic bytes to visible, harmless Unicode characters:

It is purely cosmetic — Ġ is literally the space character wearing a hat. But it carries real information: the space is part of the token. Ġcasa (" casa", preceded by a space) and casa (glued to the previous word, as in "minhacasa") are different tokens, with different IDs and different embeddings.

4 The math (there is little, but it is what matters)

BPE training is a sequence of greedy choices. At each step t, with C the current corpus and P the set of neighboring token pairs:

That pair is merged into a new token ab, the corpus is rewritten, and the pair enters the merges.txt list at position t. Repeat until |V| = 50,257.

Application (what runs when you type)

Tokenizing a new text does not redo the counts. It only replays the merges, in the same order in which they were learned:

And GPT-2's final vocabulary breaks down like this:

5 GPT-2 vs RoBERTa vs GPT-1: three different tokenizers

This is a point that confuses many people: RoBERTa uses exactly the same algorithm as GPT-2 — byte-level BPE, 50,000 merges. It was an explicit decision by the RoBERTa authors, who swapped BERT's WordPiece for GPT-2's BPE. But the special tokens and the corpus are different, so the IDs do not match.

RoBERTa's 8 extra tokens (50,265 vs 50,257) are precisely the special ones it needs and GPT-2 does not: <pad> to fill batches of different lengths, <mask> for its training objective (chapter 10), and <s>/</s> to mark start and end — and <s> later becomes the vector that carries the summary of the whole sentence for classification (chapter 12).

6 The code, line by line

# tokenizar_na_mao.py — GPT-2's BPE without magic
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")

texto = "O Brasil é gigante pela própria natureza."

# 1) text -> list of IDs (what the model actually receives)
ids = tok.encode(texto)
print(ids)
#    [46, 8290, 2634, 20692, 12427, 27638, 355, 1050, 16864, 3450, 46844, 13]

# 2) each ID back to the piece of text it represents
for i in ids:
    print(f"{i:>6}  {tok.decode([i])!r:<12}  {tok.convert_ids_to_tokens([i])[0]}")
#    the 3rd field shows the raw form, with the Ġ for the space

# 3) the metric that matters: how many tokens per word?
palavras = len(texto.split())
print(f"{len(ids)} tokens para {palavras} palavras = {len(ids)/palavras:.2f} tokens/palavra")

The output of line 3 is the moment it clicks. In English, that number sits near 1.3. In Portuguese, near 2.4. In other words: the same text costs almost twice as many tokens in Portuguese — twice the memory, twice the time, half the useful context, and (in paid APIs) twice the price. It is not that the model "dislikes" Portuguese; it is that merges.txt never saw enough Portuguese to create good pieces.

7 In the lab

cd GPT-2
.venv\Scripts\python.exe lab\01_tokenizer.py

This script does four things. Watch each one:

1. Breaks a Portuguese sentence down token by token, showing ID, text and the raw form with Ġ. Look for where the words split — almost always at the suffix.
2. Compares the same idea in English and Portuguese and prints the tokens/word ratio for both. That is the measure of the corpus bias.
3. Passes the same text through the three tokenizers (GPT-1, GPT-2, RoBERTa) and shows that the IDs bear no relation to one another. This proves, in practice, that you cannot mix one model's tokenizer with another model's weights.
4. Tests pathological cases: emoji, large number, proper name, double space. The most revealing result is the numbers: " 2024" costs 1 token (it was a frequent year on the 2018 internet), but " 1234567" becomes 3 tokens, split into 123+45+67 — pieces with no arithmetic meaning whatsoever. The model never sees the whole number. That is literally why models of this generation got simple sums wrong: they do not do arithmetic on numbers, but on arbitrary fragments of digits chosen by text frequency.

Now the text is a list of integers. But an integer has no meaning — ID 41 is not "more" or "less" than 40.
In chapter 02 these integers become vectors, and that is where meaning is born.