OpenAI's original
code

1 What is in repos/

We cloned the primary sources. They are not for running — they are for reading. After twelve chapters, you have the vocabulary to understand every line, and reading the original has a value that no second-hand explanation replaces.

2 The whole block, in the original

Here is the block function from gpt-2/src/model.py, not a line altered:

def block(x, scope, *, past, hparams):
    with tf.variable_scope(scope):
        nx = x.shape[-1].value
        a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams)
        x = x + a
        m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams)
        x = x + m
        return x, present

Read slowly and recognize everything:

• x = x + a and x = x + m — the residual connections (chapter 05). The x is never destroyed, only added to. The blackboard.
• norm(x, 'ln_1') inside the call to attn, and the sum outside — this is Pre-LN (chapter 05, section 4). The difference that allowed 48 layers is in this exact order.
• nx*4 — the MLP's 4× expansion factor, right there, literal.
• present — this layer's K and V, returned to be cached. The KV cache (chapter 04) was already in the original 2019 code.

Seven lines. It is the entire Transformer block.

3 The causal mask, in the original

def attention_mask(nd, ns, *, dtype):
    """1's in the lower triangle, counting from the lower right corner.
    Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't
    produce garbage on TPUs."""
    i = tf.range(nd)[:,None]
    j = tf.range(ns)
    m = i >= j - ns + nd
    return tf.cast(m, dtype)

def mask_attn_weights(w):
    _, _, nd, ns = shape_list(w)
    b = attention_mask(nd, ns, dtype=w.dtype)
    b = tf.reshape(b, [1, 1, nd, ns])
    w = w*b - tf.cast(1e10, w.dtype)*(1-b)
    return w

4 Three findings from reading the original

Finding 1: the GELU is the tanh approximation, written by hand

def gelu(x):
    return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3))))

The exact GELU uses the cumulative distribution function of the normal, which is expensive. This is the hyperbolic-tangent approximation — and that 0.044715 is an empirically tuned constant. It is in the original code, written by hand. If you use PyTorch's nn.GELU() with no arguments, you get the exact version, slightly different from this one.

Finding 2: the three matrices Q, K, V really are a single one

c = conv1d(x, 'c_attn', n_state*3)
q, k, v = map(split_heads, tf.split(c, 3, axis=2))

A single projection to 3 × 768 = 2304, then a cut into three. Exactly what chapter 03 described as an implementation detail — and here is the evidence.

The name conv1d is misleading: read the implementation and you will see it is an ordinary matmul. It is a kernel-1 convolution, which is mathematically identical to a dense layer. A convention inherited from the era.

Finding 3: the KV cache, in two lines

present = tf.stack([k, v], axis=1)
if past is not None:
    pk, pv = tf.unstack(past, axis=1)
    k = tf.concat([pk, k], axis=-2)
    v = tf.concat([pv, v], axis=-2)

Stack the new K and V, concatenate with the old ones. That is the whole optimization which, in lab 04, you measured to be worth a ~1.7× gain in generation — and which grows to orders of magnitude on long texts.

And it is only possible because the model is causal (chapter 11). In an encoder, there would be nothing to cache.

5 The tokenizer: GPT-3-Encoder

Despite the name, it is the GPT-2 tokenizer — GPT-3 reused the same one. And it is useful for two reasons.

First, the vocabulary files are there, in plain text:

• encoder.json — the token → ID map, all 50,257 of them.
• vocab.bpe — the 50,000 merges, in the order they were learned. The first line is the first merge of the BPE training.

You can open these files in an editor. It is the merges.txt of chapter 01, and seeing the first lines is seeing the statistics of English.

Second, it runs — in Node, which you have. It is useful when you need to count tokens in the browser or in a JavaScript service, without loading Python.

6 The gpt-3 repository: data, not code

There is no implementation — GPT-3 was never open-sourced. But there are valuable things:

• 175b_samples.jsonl — raw outputs from the 175-billion model. Compare with what gpt2-xl produces and you see, with your own eyes, what 112× more parameters buy.
• overlap_frequency.md — contamination analysis: how much of the evaluation sets leaked into the training corpus. It is an honest and rare document, and a good reminder that LLM benchmarks are more fragile than they seem.
• dataset_statistics/ — composition of the corpus.

7 In the lab

.venv\Scripts\python.exe lab\10_original_code.py

1. Shows excerpts of the original model.py side by side with the equivalent PyTorch implementation, so you can see the line-by-line correspondence.
2. Opens vocab.bpe and shows the first 20 merges the BPE learned. The first ones are the most frequent in English.
3. Confirms that the JS tokenizer and the Python one produce the same IDs — runs Encoder.js via Node and compares.
4. Extracts GPT-3 samples from 175b_samples.jsonl and shows them next to a gpt2-xl generation on the same theme.

You have read the source. Chapter 14 closes the lab with the last practical piece:
how to serve these models over HTTP — and what TGI and TEI do that our server does not.