Map of the course
and the lab

1 Why this course exists

This course takes apart GPT-2 and RoBERTa — the seven models the openai-community organization published on Hugging Face. They are not a random list: they are a historical cross-section. Between 2018 and 2019, the entire field of language processing split into two branches, and these seven files are exactly the record of that moment.

On one side, the decoder branch: GPT-1 → GPT-2 small → medium → large → XL. This is the lineage that generates text, one word at a time, looking only backward. It would lead to GPT-3, to ChatGPT and to everything that came after.

On the other side, the encoder branch: BERT → RoBERTa. This is the lineage that understands text, looking at the whole input at once, forward and backward. It remains widely used for classification, search and information extraction.

And there is a delightful bridge between the two branches: the two roberta-*-openai-detector models. They are RoBERTas trained specifically to detect text written by GPT-2. OpenAI itself built the hunter alongside the prey. Studying the two side by side is not just convenient — it is the best possible lesson on the difference between generating and understanding.

2 The seven models, in numbers

Notice two things that already tell a story:

• GPT-2 small and RoBERTa base have practically the same size (124 M vs 125 M) and the same shape (12 layers, 768 dimensions, 12 heads). That is no coincidence: both copied BERT's "base" configuration. They are structural twins that do opposite things. It is the perfect controlled experiment.
• The GPT-2 family grows by changing three numbers: layers, width and heads. The recipe does not change, only the scale. You will see with your own eyes, in chapter 06, what 12× more parameters buy.

3 The track, chapter by chapter

The order matters. Each chapter uses the vocabulary built in the previous one. Do not skip — especially from 01 to 05, which are the foundation. Words such as token, layer and model carry their machine-learning meaning here, which is not always the same as in the rest of IT; the glossary page, which comes right after this map, shows both sides.

Before Part I — Reference page

Part I — The foundation (how a Transformer works)

Part II — The GPT family (generating)

Part III — The BERT family (understanding)

Part IV — Putting it on your machine

Part V — Where this path stops

4 Where everything lives on your machine

# the repository root, wherever you clone it
├── .venv\                  # Python 3.14 + torch 2.14 (CPU) + transformers 5
├── models\hf\              # the 7 models, 13.2 GB in safetensors
├── repos\
│   ├── gpt-2\              # OpenAI's original code, in TensorFlow (to READ)
│   ├── GPT-3-Encoder\      # tokenizer in JS + vocab.bpe + encoder.json
│   └── gpt-3\              # samples from the 175B model and statistics
├── lab\                    # << the 11 experiments for you to run
├── server\app.py           # << HTTP API compatible with TGI and TEI, and the workbench
├── model\                  # the ELLM built in the next experiment (Antivision)
├── tools\                  # download, PDF generation, verification
├── src\conceitos_gpt-1-gpt-2\  # HTML source of each chapter (eng\ = English version)
├── src\conceitos_antevisao\    # the theory of the ELLM, Antivision experiment
├── pdfs\gpt-1-gpt-2\          # << the 17 chapters and the glossary (eng\ in English)
└── out\                    # experiment outputs

5 What exists beyond what we use

You will run into these tools out there. It is worth knowing what each one is for, even without using them here.

Note the third column: only transformers and TEI cover both branches of the tree. It is a real asymmetry of the ecosystem — the popular "run AI locally" tools were built for models that generate, and ignore the ones that understand.

6 How to study this

Every chapter from 01 onward always has the same structure, and it was designed for you to read with the computer open next to you:

1. Intuition — the idea in plain language, without a single symbol.
2. Analogy — a mental image to hold on to the idea.
3. Math — the formula, with every symbol explained one by one. I never throw in an equation without saying what each letter is.
4. Code — the same thing in Python, commented line by line.
5. In the lab — the exact script to run and what to watch for in the output.
6. Exercise — one thing for you to change and predict the result before running.

Next: the glossary page, then chapter 01 — Tokenization and BPE.
Start chapter 01 even if you think you already know it: BPE has details, such as the space that becomes part of the token, that change how the model behaves.