What it costs to reach GPT-3 scale, why VRAM is almost never the limit, and the formula that says which size to train with the money you have.
The GPT-3 weights were never published. The openai/gpt-3 repository has text samples and corpus statistics — no weights. There is no such thing as "downloading GPT-3".
And architecturally it is, in essence, a bigger GPT-2: same blocks, same causal attention, with alternating sparse-attention layers. What separates the two is scale, not invention.
So "reaching GPT-3" means climbing size steps. And the paper defines eight of them, not one — the 175 B is the last, not the only one.
Chinchilla (Hoffmann et al., 2022) measured the relationship between parameters, data and compute. The rule of thumb: ~20 training tokens per parameter.
The cost in floating-point operations to train
The factor 6 comes from two multiplications per parameter in the forward pass and four in the backward pass. Substituting
Given a budget, there is one optimal size. Training bigger leaves the model undertrained — it has capacity that was never filled. Training smaller wastes compute on something that has already converged.
"I'll take the biggest model that fits in VRAM" is the wrong heuristic, and it is the most common one.
On a rented RTX 5090 (32 GB, US$ 0.678/h), at 60 effective TFLOP/s:
| Model | Params | Tokens | Text | VRAM | Hours | Cost |
|---|---|---|---|---|---|---|
our mini ✓ | 7 M | 0.1 B | 1 GB | 0.2 GB | — | — |
our grande | 97.5 M | 1.9 B | 7 GB | 2.1 GB | 5.3 | US$ 2 |
| GPT-3 Small | 125 M | 2.5 B | 9 GB | 2.7 GB | 8.7 | US$ 4 |
| GPT-3 Medium | 350 M | 7.0 B | 26 GB | 7.6 GB | 68 | US$ 31 |
| GPT-3 Large | 760 M | 15.2 B | 57 GB | 16.6 GB | 321 | US$ 145 |
| GPT-3 XL | 1.3 B | 26 B | 97 GB | 28.3 GB | 939 | US$ 424 |
| GPT-3 2.7B | 2.7 B | 54 B | 201 GB | 58.8 GB ✗ | 4,050 | US$ 1,831 |
| GPT-3 175B | 175 B | 3.5 T | 13 TB | 3,814 GB ✗ | 1,942 years | US$ 7.7 M |
VRAM is almost never the limit. Up to GPT-3 Large fits in the 32 GB. Only from 2.7 B on does the card become too small — and at that point the problem stops being renting one and becomes parallelizing across several, which is a different kind of engineering.
The limit is the corpus. We have 310.6 M tokens. GPT-3 Small asks for 2.5 billion: 8× more text than we have. No card solves that.
The 175 B is not a question of your money. Almost two thousand years on one card. That was done with thousands of GPUs in parallel for months — another category of undertaking, not the next step.
Training requires more than the weights. With AdamW: the weights, the gradients, and two moments per parameter. In fp32 that is 16 bytes per parameter; with mixed bf16, ~18 counting the master copy. We add ~30% for activations:
A 1 B model asks for ~24 GB just for the optimizer state. That is why the 2.7 B row overflows a 32 GB card — not because of the weights, which are 5.4 GB.
The 60 effective TFLOP/s are an estimate, not a measurement. We calibrated against a real data point: nanoGPT trains the 124 M GPT-2 on 8×A100 at ~870 k tok/s, or ~110 k per card. Our formula gives ~117 k for an equivalent model — it matches.
On small models, it lies upward. A 3 M parameter model saturates no card at all: the bottleneck becomes kernel launches and data feeding, not computation. That is why the table marks with ? the rows that hit the conservative ceiling of 400 k tok/s.
The solution is not to estimate: cloud/calibrate.py runs a few real steps at each size, measures, and only then recommends. It costs ~4 cents.
A small budget in this rental range buys on the order of 7.4 hours of one card. Applying
| Effective throughput | Budget in FLOPs | Optimal size |
|---|---|---|
| 40 TFLOP/s (conservative) | 1.07 × 10¹⁸ | 94 M |
| 60 TFLOP/s (optimistic) | 1.60 × 10¹⁸ | 115 M |
Our grande preset has 97.5 M. GPT-3 Small has 125 M. Seven hours of a single card buy exactly that range — trained to the optimal point, not halfway.
What it requires is ~2 billion tokens, or ~8 GB of text. That is the missing step, and it is one of data, not money.
Three kinds of waste appear when preparing a rental — all found by reading the code before running it.
| Leak | Cause | Cost |
|---|---|---|
| Training on the rented machine's CPU | the training script never moves the model to the GPU | every hour contracted |
| Serial tokenization | tok.encode one document at a time runs on one thread | ~35% of the budget |
| Unrecoverable weights | checkpoint saved with CUDA tensors | all the work |
The second is the most instructive. Swapping tok.encode(one_text) for tok(list_of_texts) triggers the parallelism of the Rust backend:
one by one : 0.80 M chars/s batched : 6.35 M chars/s -> 7.9x faster 8 GB corpus: 178.2 min one by one vs 22.5 min batched
35% of the budget, in one line of code. And on a machine billed by the hour, data preparation costs as much as training — the GPU sits idle while the CPU tokenizes.
destroy stops the billing. stop pauses the GPU and keeps charging for the disk.| # | Step | Cost | Unlocks |
|---|---|---|---|
| 1 | train pequeno to convergence | free (4 h of CPU) | first serious model |
| 2 | corpus to ~10 GB | download time | GPT-3 Small becomes viable |
| 3 | rent and train grande | ~US$ 2 | scale of the base GPT-2 |
| 4 | GPT-3 Small with our tokenizer | ~US$ 4 | scale parity |
| 5 | GPT-3 Medium | ~US$ 31 | sensible ceiling for a personal project |
Reaching GPT-3 Small scale costs less than eight hours of a single GPU. What is missing is not card time: it is text. That is why the priority is the corpus, and why chapter 01 insists so much on the tokenizer — every token saved is training budget given back.
With the arithmetic done, chapter 08 shows the bench: where all of this is operated
— serving, X-raying and training, without leaving the browser.