Rafhael Oliveira · IA

Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit

The law that organises everything, the ladder with prices, and the formula for the right size for the money you have.

1 min read· chapter: 7 min· Experiment in progress · chapter updated on 12 Sept 2026

Landed here from LinkedIn? This is chapter 07 of the experiment Antivision · Latent Rehearsal. It reads on its own; if you want the beginning, it is chapter 00.

Reaching the scale of GPT-3 Small costs less than eight hours of a single GPU. What is missing is not the card. It is text. First, undo a misunderstanding: GPT-3's weights were never published. There is no 'downloading GPT-3'. And architecturally it is a bigger GPT-2. What separates the two is scale, not invention. There is a law that organises everything, a formula that says which size to train with the money you have, and a conclusion that changes the project's priority: VRAM is almost never the limit.

The three points

  1. GPT-3 is GPT-2 at scale: same blocks, same causal attention, alternating sparse attention. The invention was 2019; 2020 was money.
  2. The scaling law says model and data grow together. Training a big model on little text is waste.
  3. The conclusion that guides the project: the priority is corpus. Every token saved in the tokenizer is training budget given back.

From the chapter, the beginning

What it costs to reach GPT-3 scale, why VRAM is almost never the limit, and the formula that says which size to train with the money you have.

The GPT-3 weights were never published. The openai/gpt-3 repository has text samples and corpus statistics — no weights. There is no such thing as "downloading GPT-3".

And architecturally it is, in essence, a bigger GPT-2: same blocks, same causal attention, with alternating sparse-attention layers. What separates the two is scale, not invention.

So "reaching GPT-3" means climbing size steps. And the paper defines eight of them, not one — the 175 B is the last, not the only one.

Keep reading chapter 07 →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 7 sections · 7 min
  1. First, undo a misunderstanding
  2. The law that organizes everything
  3. The ladder, with prices
  4. Where each number comes from
  5. What seven hours of a card buy
  6. Where the money leaks
  7. The path
All chapters of the experiment · Antivision · Latent Rehearsal
  1. 00 · Antivision: an ELLM, a language model that rehearses in latent space before speaking
  2. 01 · The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table shows
  3. 02 · Two architectures, one file: the fair duel between 2019 and what came after
  4. 03 · The geometry of the embedding: training starts with a perfect space and ruins it
  5. 04 · The lineage of collapse: five answers from computer vision, each measured on text
  6. 05 · The rudder and the world model: three versions of the same question, each measured against the last
  7. 06 · Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errors
  8. 07 · Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limit
  9. 08 · The workbench: serve, x-ray and train from the browser
  10. 09 · The hierarchical architecture: the whole project aligned to the four patterns of the world model
  11. 10 · What JEPA is, according to LeCun: two talks read concept by concept, and where each one applies
  12. 11 · Memory and horizon: two claims from LeCun that became experiments, and both hold for text
  13. 12 · Learned costs and inference modes: the three searches and the two sources of energy, measured
  14. 13 · The corrections from LeCun's talks: failure, passage, flag, measurement
  15. 14 · Consulted memory: a bank of abstract states read by attention
  16. 15 · What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers
  17. 16 · The minimum was off the list: the day I measured my own search by energy
  18. 17 · ELLM: the right name for a language model that predicts in latent space and decides by energy
  19. 18 · Searching better does not fix the energy: three searches, the same text
  20. 19 · The inversion that did not repeat: the world model's cost appears once the ruler stops swinging
  21. 20 · What changed on 17/09: two new chapters and two corrections

Before this: the GPT-2 course

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →