Why GPT-2 sees neither letters nor words, and what that costs in Portuguese

GPT-2 has never read a word of yours. Not even a letter. It reads pieces. A neural network is a matrix-multiplying machine. Before any intelligence, your text has to become numbers. That translation is not learned: it is a fixed table, decided once and frozen forever. Three generations of researchers tried to answer 'numbers how?'. The first two failed in specific ways. The third won for a reason that explains why GPT-2 writes 'tion' with ease and struggles with 'ção'.

The three points

1. Character by character is too long; word by word does not know what to do with a new word. BPE sits in between and learns the pieces that appear often.
2. GPT-2's move was BPE over bytes: there is never an unknown token, because everything is a byte.
3. GPT-1, GPT-2 and RoBERTa have three different tokenizers, and that changes even what 'perplexity' means between them.

From the chapter, the beginning

How a text becomes a list of numbers — and why the model sees neither letters nor words, but pieces.

A neural network is, at bottom, a matrix-multiplying machine. Matrices are made of numbers. So the first thing that must happen to your text is a translation: letters become numbers. This step is called tokenization, and it happens before the model — it is not learned along with it; it is a fixed table, decided once and frozen forever.

The word token is older than language models, and has other meanings in IT: the credential a server issues after login, such as a JWT; the unit a compiler's lexical analyser splits off, such as a keyword; the asset recorded in a blockchain contract. In this course, a token is always a piece of text — a word, part of a word or a byte — with an integer attached. The course glossary makes this distinction for every term that has more than one meaning.

The question is: numbers for what? There are three historical answers, and understanding why the first two failed is the only way to understand why the third won.