Two architectures,
one file

1 The problem of comparing architectures

A new piece is presented under the conditions in which it was developed: another scale, other data, another budget. A gain measured there does not automatically carry over here. That is why this chapter redoes the comparison under our conditions — same data, same seed, same number of steps — knowing that a fair comparison is laborious and expensive.

There are three traps, and all of them were deliberately avoided here:

2 The four swaps

Position: learned wpe → RoPE

GPT-2 has a wpe table with exactly 1024 rows — one per possible position. It is a learned vector, added to the token embedding. It works, and it has two hard consequences.

The first is the wall. We measured it directly:

contexto  512  → 29,6 ms/token
contexto 1000  → 34,6 ms/token
contexto 1024  → PAREDE: posição 1024 não existe na wpe (ela tem 1024 linhas: 0 a 1023)

It is not a configuration limit. It is a matrix that runs out. To extend the context, new rows have to be created — rows that were never trained.

RoPE (Su et al., 2021) solves it by construction: instead of adding a position vector, it rotates the query and key vectors by an angle proportional to the position. Since the dot product between two rotated vectors depends only on the difference of the angles, attention comes to see relative distance — and there is no table to run out.

Cost in parameters: the wpe of mini has 1024 × 256 = 262,144 weights. RoPE has zero — it is a computed rotation, not a learned one.

Normalization: LayerNorm → RMSNorm

LayerNorm does two things: it centers (subtracts the mean) and scales (divides by the standard deviation). RMSNorm (Zhang & Sennrich, 2019) observed that centering contributes almost nothing, and removed it:

Gone is the mean subtraction and gone is the shift parameter b. Less arithmetic, half the normalization parameters, and no measured loss in the literature.

MLP: GELU 4× → SwiGLU

GPT-2's block expands to 4d, applies GELU and comes back. SwiGLU (Shazeer, 2020) uses two projections in parallel: one becomes the signal, the other becomes a gate, and they multiply.

class SwiGLU(nn.Module):
    def forward(self, x):
        return self.w_down(F.silu(self.w_portao(x)) * self.w_cima(x))

That is three matrices instead of two, so the inner dimension shrinks (from 4d to ⅔·4d) to keep the count. The gate lets the network learn when to let the signal through, rather than only how much.

Biases: all → none

Llama removed the b term from every linear layer. Fewer parameters, and in practice no loss. We also added QK-norm (Dehghani et al., 2023) — normalizing query and key before the dot product, which prevents attention logits from blowing up in long training runs.

3 The duel

mini size (7.4 M), 700 steps, 5.7 M tokens, same seed. Only the architecture changes.

−24% perplexity, and 100 fewer steps to reach the best point. On the four bits/character measurements:

A correction the method caught

During the duel, the classic model's validation rose between steps 649 and 699 (5.539 → 5.577). The immediate reading was "it started overfitting". It was wrong.

With 5.7 M tokens out of a 60.3 M corpus, the model saw 0.09 of an epoch — no token was seen twice. Overfitting is impossible by construction. What rose was evaluation noise added to the closing of the learning-rate cosine.

4 The price of each piece, in parameters

The modern one is slightly smaller and still wins. It is the strongest argument in the table: it is not more capacity, it is better use of the same.

5 How to read the code

Everything in model/architecture.py, with each modern piece written immediately next to the classic one it replaces:

def normalizador(cfg, d):
    return RMSNorm(d) if cfg.moderno else nn.LayerNorm(d)

# in the Transformer:
if cfg.moderno:
    self.wpe = None                        # RoPE goes into attention
    rope = RoPE(cfg.d_k, cfg.n_ctx, cfg.rope_base)
else:
    self.wpe = nn.Embedding(cfg.n_ctx, cfg.d_model)
    rope = None

Reading that file top to bottom is seeing the difference between 2019 and today in ~400 lines. That was the goal — more than having both options, having both legible side by side.

With the architecture settled, chapter 03 looks inside the embedding table —
and finds a problem that none of these modern pieces fixes.