Rafhael Oliveira · IA

Two architectures, one file: the fair duel between 2019 and what came after

RoPE, RMSNorm and SwiGLU next to the pieces they replace, with the same corpus, the same budget and the same seed.

1 min read· chapter: 6 min· Experiment in progress · chapter updated on 12 Sept 2026

Landed here from LinkedIn? This is chapter 02 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.

A new piece is presented under the conditions of whoever built it. We redid the comparison under ours: same data, same seed, same budget. It is not dishonesty. It is that the fair comparison is hard and expensive. The traps are three: different implementations, different parameter counts, different budgets. We wrote RoPE, RMSNorm and SwiGLU in the same file as the 2019 pieces they replace, with the same training loop, the same frozen corpus, the same 700 steps and the same seed. 17.1 million parameters against 16.9. Perplexity: 254 against 193.

The three points

  1. The four swaps: rotary position instead of a table, normalisation with gain only, MLP with three matrices, linears without bias. The position table disappears: 393 thousand parameters fewer.
  2. The duel: validation 5.539 against 5.262; perplexity 254 against 193; bits per character 2.342 against 2.283 on average.
  3. What it does not prove: one seed, one size, 0.09 epoch. The four pieces changed together; isolating each is four runs that are on the list, not in the result.

From the chapter, the beginning

RoPE, RMSNorm and SwiGLU written next to the 2019 pieces they replace — and the duel with the same corpus, the same budget and the same seed.

A new piece is presented under the conditions in which it was developed: another scale, other data, another budget. A gain measured there does not automatically carry over here. That is why this chapter redoes the comparison under our conditions — same data, same seed, same number of steps — knowing that a fair comparison is laborious and expensive.

Keep reading chapter 02 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 5 sections · 6 min
  1. The problem of comparing architectures
  2. The four swaps
  3. The duel
  4. The price of each piece, in parameters
  5. How to read the code
All chapters of the experiment · Antivision · Latent Rehearsal
  1. 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
  2. 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
  3. 02 · Two architectures, one file: the fair duel between 2019 and what came after
  4. 03 · The geometry of the embedding: training starts with a perfect space and ruins it
  5. 04 · The lineage of collapse: five answers from computer vision, each measured on text
  6. 05 · The rudder and the world model: three versions of the same question, each measured against the last
  7. 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
  8. 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
  9. 08 · The workbench: serve, x-ray and train from the browser
  10. 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
  11. 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
  12. 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
  13. 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
  14. 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
  15. 14 · Consulted memory: a bank of abstract states read by attention
  16. 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
  17. 16 · The minimum was off the list: the day I measured my own search by energy
  18. 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
  19. 18 · Searching better does not fix the energy: three searches, the same text
  20. 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
  21. 20 · What changed on 17/09: two new chapters and two corrections

Before this: the GPT-2 course

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →