Pretraining, loss
and perplexity

1 The task: predict the next word. That's all.

GPT-2 was never taught to answer questions, summarize texts, translate or write code. It was trained on a single task, repeated billions of times:

This is called causal language modeling (CLM). Formally, the model learns the probability of an entire sequence, decomposed by the chain rule:

The hidden efficiency

Each sequence of n tokens produces n simultaneous training examples, not one. A text of 1024 tokens generates 1024 predictions, all charged at once, in a single pass. That is why it is called self-supervision: the labels are the text itself, shifted by one position. No human annotated anything.

One sequence, many examples:

input    The  capital  of    France  is
target   capital  of   France  is    Paris
         ^       ^      ^      ^      ^
         5 predictions, 5 losses, 1 pass through the model

Alignment in code: logits[:, :-1]  against  labels[:, 1:]
(drops the last prediction, which has no target, and the first
token, which is predicted by nobody)

2 The loss: cross-entropy

How is a mistake punished? With the negative logarithm of the probability assigned to the correct token:

Why the logarithm

Three reasons, and all three are worth understanding:

• The punishment is asymmetric and grows without bound. If the model gave 90% to the right token, the loss is −log(0.9) = 0.105 — almost nothing. If it gave 1%, it is 4.6. If it gave 0.001%, it is 11.5. Being confident and wrong costs a great deal. That is what teaches the model to be calibrated, not just accurate.
• It turns a product into a sum. The probability of a sequence is a product of thousands of small numbers — that becomes zero in floating point almost immediately. In log space, it becomes a sum, numerically stable.
• It has a physical meaning. Cross-entropy is measured in nats (or bits, if log base 2) and answers: "how many bits do I need, on average, to transmit the next token, using the model as a compressor?" Training a language model is literally training a text compressor.

3 Perplexity: the loss in human units

The loss is useful but not very intuitive. Perplexity is simply its exponential:

And it has a direct and very useful reading:

References to calibrate your intuition

4 How GPT-2 was trained, in practice

5 The code

# perplexity of a text, the right way
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
mod = AutoModelForCausalLM.from_pretrained("openai-community/gpt2").eval()

def perplexidade(texto):
    ids = tok(texto, return_tensors="pt", truncation=True, max_length=1024)
    # by passing labels=input_ids, transformers already does the
    # 1-position shift and computes the mean cross-entropy.
    with torch.no_grad():
        perda = mod(**ids, labels=ids["input_ids"]).loss
    return torch.exp(perda).item()

print(perplexidade("The capital of France is Paris."))     # low
print(perplexidade("Colorless green ideas sleep furiously.")) # high

6 In the lab

.venv\Scripts\python.exe lab\08_perplexity.py

1. Measures the perplexity of several kinds of text — natural English, Portuguese, shuffled text, random text, code, and Chomsky's famous sentence. The order of the results says a lot about what the model learned.
2. Compares the five models on the same text. Here the scaling shows up clean and monotonic, unlike the anecdote in chapter 06. This is the honest measurement.
3. Shows the surprise token by token, highlighting where the model was startled. It is the most direct view of what the model "knows".
4. Demonstrates that GPT-2 is not an assistant, comparing a prompt in the form of a question with one in the form of a beginning of a text.
5. Measures perplexity in Portuguese and separates the two effects that inflate it: not knowing the language and tokenizer fragmentation.

7 Two lab results worth the whole chapter

First: scaling behaves when measured properly

Monotonic at every step. No inversion, no surprise. Compare with chapter 06, where large "missed" a question that medium "got right".

The models are the same in both experiments. What changed was the metric. A single sample measures noise; perplexity measures the distribution. Keep this pair of results — it is the best methodology lesson in this course.

Also note the jump GPT-1 → GPT-2: from 46 to 21.7 with practically the same number of parameters (117 M vs 124 M). That was not scale. It was the better corpus, the better tokenizer and Pre-LN. It is the subject of the next chapter.

Second: random garbage is easier than a grammatical sentence with no meaning

Pure noise comes out eight times cheaper than shuffled English, and thirteen times cheaper than Chomsky's sentence. This seems strange until you understand what perplexity really measures.

Methodological caveat: Chomsky's sentence has only 7 tokens. Averages over tiny samples are unstable, and a single improbable token dominates the result. Treat that number as illustrative. General rule: distrust perplexity computed on short texts.

End of Part II. You understand the entire GPT family: how it is assembled, how it chooses words, and how it is measured.
Part III crosses over to the other branch of the tree — and chapter 09 first settles the account with GPT-1.