GPT-2 end to end,
and what scale buys

1 The whole model, top to bottom

You already know all the pieces. Here is the assembly, with nothing omitted:

TEXT    "The capital of France is"
   |
   |  BPE tokenizer (chapter 01)              — not part of the model
   v
IDs     [464, 3139, 286, 4881, 318]                        (5,)
   |
   |  wte[IDs]  +  wpe[0..4]                 (chapter 02)
   v
h_0                                                        (5, 768)
   |
   +-- BLOCK  1 ---+                                                 .
   |   x = x + attn(ln_1(x))   communication (chapters 03, 04)       .
   |   x = x + mlp (ln_2(x))   computation   (chapter 05)            .
   +---------------+                                                 .
   |                                                                 .
   |  ... 12 identical blocks, shape always (5, 768) ...             .
   |                                                                 .
   +-- BLOCK 12 ---+
   |
   v
h_12                                                       (5, 768)
   |
   |  ln_f   — final LayerNorm (exists only because of Pre-LN)
   v
   |  @ wte.T   — weight tying (chapter 02, section 6)
   v
LOGITS                                                   (5, 50257)
   |
   |  we take ONLY the last row: the prediction of the next token
   v
logits[-1]                                                 (50257,)
   |
   |  softmax + sampling strategy             (chapter 07)
   v
NEXT TOKEN  ->  goes back to the top, now with 6 tokens

2 The whole family in one table

The four GPT-2s are the same model. No new piece, no different trick. Only three numbers change.

They all share: a vocabulary of 50,257, a context of 1024, byte-level BPE, Pre-LN, GELU, 4× expansion in the MLP, weight tying. The training was the same too — the same WebText, the same 40 GB of text. The only experimental variable was size.

That was, in fact, the thesis of the GPT-2 paper ("Language Models are Unsupervised Multitask Learners", 2019): keep everything fixed and grow, to show that new capabilities emerge from scale alone.

The counting formula

The term that matters is L · 12d². Since d is squared, doubling the width quadruples the cost of that block. That is why models grow more in depth than in width: it is cheaper.

3 What scale really buys

The intuitive answer ("bigger models write better") does not hold up in the lab.

In this course's initial test, I asked the same thing of all four models, in greedy mode (always the most likely word). The prompt was "The capital of France is":

So what really improves?

When you measure properly, over many examples, three things improve consistently with scale:

1. Perplexity. The measure of "how surprised the model is by real text" drops cleanly and predictably. You will measure this in lab 08.
2. Long-range coherence. The small loses the thread within 2–3 sentences; the XL sustains a paragraph. It is what the logit lens of chapter 05 suggested: the small's problem is not ignoring the fact, it is failing to sustain it all the way to the output.
3. In-context learning. This is the genuinely emergent capability: giving examples in the prompt and having the model follow the pattern with no training at all. See the measurement below.

The measurement that shows scale clearly

The same prompt as in the previous table, now with three examples before it:

Portugal -> Lisbon
Japan -> Tokyo
Italy -> Rome
France ->

Look at the last column. The runners-up are Marseille, Lyon, Nice — other French cities. The model did not merely get it right: it understood that the task was "country → city of that country" and started erring within the right category. That is the pattern being captured, not a memorized correspondence.

And note: no weight was updated. The only difference between 3.2% and 38.5% was the text that came before the question. Taken to GPT-3 (175 billion parameters), this capability became so strong that it made fine-tuning unnecessary for most tasks — and it is the reason "prompt engineering" became a discipline.

And what does not improve

• Arithmetic. The cause is the tokenizer (chapter 01), not the size. No GPT-2 adds well.
• Reliable factuality. They all hallucinate. The XL hallucinates more fluently, which is worse, not better.
• Portuguese. The corpus was English. Scale does not create data that did not exist.
• Context. Still 1024 tokens in all of them — it is the wpe, not the capability.

4 The cost of running, measured

On an ordinary 8-core desktop CPU, in fp32 and without a GPU, measured in practice:

The rule of thumb: fp32 costs 4 bytes per parameter. A 1.5 B model takes up 6 GB in weights alone. That is why quantization exists — in 8 bits, the same model fits in 1.5 GB, and in 4 bits, in 800 MB. Quality drops a little and the practical reach changes completely.

5 In the lab

.venv\Scripts\python.exe lab\06_scale.py

1. Checks the architecture of each model by reading the real config.json, and verifies the parameter-count formula against the true number.
2. Runs the same prompt on the five models, greedy, and shows the table above live on your machine.
3. Measures the probability each model assigns to the right answer — not just what it generated. This is far more informative than the single sample: you will see that the large "almost" got it right.
4. Tests in-context learning with examples in the prompt, comparing small and XL. It is where scale shows up most clearly.
5. Times everything and builds the cost table for your machine.

The model produces a probability distribution over 50,257 tokens. Turning that into text is your decision, not its.
Chapter 07 shows that this decision changes the result more than switching models does.