GPT-1 and GPT-2, taken apart
Seven models from 2018 and 2019, opened line by line: how text becomes numbers, what attention does, what scale buys, and the single line of code that separates GPT from BERT. Everything runs on an ordinary machine, no GPU.
A course: each chapter teaches one piece of the model and proves it in a lab that runs on your machine.
For whom has never opened a model up, or uses an LLM every day and wants to know what happens between the question and the answer.
What you walk away knowing: how text becomes numbers, what attention does, what scale buys, and the single line of code that separates GPT from BERT.
Chapters
In order. Each chapter uses the vocabulary of the previous one.
- 00The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a listSeven files on Hugging Face record the exact moment language AI split in two.
- GlossaryCourse glossary: what token, layer and model mean in AI, and in the rest of IT"Token" already had other owners before AI: the compiler, the login and the blockchain.highlight
- 01Why GPT-2 sees neither letters nor words, and what that costs in PortugueseGPT-2 has never read a word of yours. Not even a letter. It reads pieces.
- 02Embeddings: from a meaningless integer to a vector that knows king and queen are relatedTo the model, cat is 4899 and dog is 5257. Feeding that into a neural network is a serious conceptual error.
- 03Attention: the single operation that replaced twenty years of recurrent networks"The trophy didn't fit in the suitcase because it was too big." What is 'it'?
- 04Twelve heads instead of one, and the single line of code that separates GPT from RoBERTaAttention is a fixed budget: every row sums to 1. If 70% goes to one word, 30% is left for everything else.
- 05Why GPT-2 needs 48 floors to answer, and what happens inside each oneWhen you send ChatGPT a question, it climbs 96 floors before it becomes an answer. In GPT-2 it was 48. Every floor does the same thing. So why so many?
- 06What scale really buys: from 124 million to 1.5 billion, measuredGPT-2 large, with 774 million parameters, got the capital of France wrong. Medium, with 355 million, got it right.
- 07Temperature, top-k and top-p: the decision that changes the text more than switching modelsThe model does not choose the next word. You do.
- 08Perplexity: the one number that says what the model learned, without anecdotesGPT-2 was never taught to answer questions. It was trained on a single task, billions of times.
- 09GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain itDo you know how GPT-1 became GPT-2? And why it happens?highlight
- 10RoBERTa: when the advance is training better what already existedFour months after GPT-1, Google took the same block and made the opposite decision.
- 11Encoder vs decoder: one line of code and every consequence that descends from itA minus-infinity sign decides whether the model writes or understands. There is no free lunch.highlight
- 12The detector: how a language model becomes a classifier, and why this one should decide nothingOpenAI built the hunter with the architecture opposite to the prey's. And since 2023 it is worse than useless.
- 13Reading GPT-2's original code: 174 lines against everything you learnedThe whole of GPT-2 fits in 174 lines of TensorFlow. And you can read every one of them.
- 14Serving models: from function call to HTTP server, and what production does on topLoading GPT-2 XL costs 7 seconds and 5.8 GB of RAM. Every single time.
- 15The lab guide: every command, every script, and everything that was measuredGPT-2 opened the path behind every LLM you use today. How far does that path go?highlight
- 16The ceiling of this path: LeCun's argument, read from inside GPT-2Every text humanity has written fits inside what a four-year-old has already seen.
The whole collection as PDFs, in your inbox
All 18 chapters, in Portuguese, English or both. Leave your e-mail and the links arrive in one send. You join the list and get new pieces as they come out — to leave, it is one click at the bottom of any e-mail.