The rules that make a comparison legitimate, and the complete catalog of this project's mistakes — including the ones that only surfaced because someone asked "compared to what?".
Recording only what worked would distort the reading of everything that follows. And there is also a practical reason: half of the best decisions in this project came from discovering we were wrong. The cosine regularizer exists because a semantic search failed. The floor metric for the compass (the first version of the rudder, which only predicted the direction of the next state) exists because a pretty measurement turned out to be empty.
This chapter records the rules and the mistakes. Whoever repeats the project will err differently — but the kind of error repeats.
Every comparison uses the same treino.bin, the same number of steps, the same batch and the same seed 1337. The duel corpus is frozen in model/data/duelo_200mb/, with its hash checked.
And the packaging script refuses to overwrite existing data:
PARADO — ja existe um conjunto de dados aqui
treino.bin 115.1 MB
Sobrescrever isso quebra a comparabilidade com todo treino ja feito.
Without that lock, a corpus swapped in the middle of a series would invalidate everything silently — the numbers would keep coming out, they just would no longer mean the same thing.
Validation oscillates between evaluations. In the duel, the classic model got worse between steps 649 and 699 with no real cause. Comparing the last step of each run compares noise.
mini-moderno-reg1-lat1-duelo is ugly and it is right: size, architecture, regularizer, compass, run. Six months later, the question "what was in that run?" is answered by the label, not by memory.
The report detects runs that did not spend the same budget and writes PARCIAL (partial) in place of the percentage difference. A run stopped at step 100 against others at 700 would show a huge number that says nothing.
This rule was born from a mistake, told in section 4. A number on its own means nothing: 0.534 is excellent or is nothing, depending on the floor.
| Mistake | Consequence | How it was caught |
|---|---|---|
| Centering the data in SigReg | the cone scored better than the Gaussian | synthetic test with a known answer |
| Reporting the compass without a floor | a gain of +0.07 presented as +0.53 | asking "compared to what?" |
| "The classic model started overfitting" | impossible diagnosis | 0.09 epoch — no token seen twice |
| "Dead tokens will pollute the top-50" | plausible hypothesis, not confirmed | measuring on 6 prompts: 0 out of 1.44 expected |
| Comparing against the wrong control | −1.27% became +1.06% when the base was swapped | building the table with both columns |
| "The latent loss is free" | claimed at step 249; it was +1.41% | best against best, at the end |
open(arquivo, "w") on the corpus | 60 MB of code erased by an error in another source | the file became 0 bytes |
| Model cache with no ceiling | server dead, os error 1455 | virtual memory filled up |
gerar_pdfs pointing at a renamed folder | generation broken silently | trying to generate the new collection |
| Training script with no GPU support | would pay for the card by the hour to train on CPU | reading the code before renting |
| Tokenizing document by document | 178 min instead of 22, for 8 GB | measuring before running in the cloud |
Told in chapter 03, but the general form is worth stating. The line was:
Z = Z - Z.mean(0, keepdim=True) # "normalize"
A routine normalization. Except that the anisotropy we wanted to measure is a shift of the origin — and centering removes shifts. The instrument erased the signal before measuring it.
Training did not complain: the loss went down, the model learned. The mistake only appeared when the meter was run against distributions whose correct answer we already knew.
The compass predicts the direction of the next state. We measure the cosine between the prediction and the real target: 0.534. For several runs we reported it as "0.534 against ~0.00 for chance".
Until someone asked what it would give with no predictor at all — just copying the current state:
| Reference | Cosine |
|---|---|
| cos( | 0.478 |
| cos( | 0.461 |
| shuffled target — true chance | 0.206 |
| the shallow compass | 0.534 |
And chance was never 0.00 — it is 0.206, because the hidden states themselves have residual anisotropy. The shallow compass was learning almost nothing beyond copying its input, and charged 1.41% of quality for it.
The numbers had been saying so all along. They were hidden behind a metric with no reference.
The fix was structural, not editorial: the floor is now computed inside the training loop and printed next to the metric, always. It is no longer possible to report one without the other.
bussola 0.064 (piso 0.598 ganho -0.534)
That line is from a freshly initialized model: the compass is worse than copying the current state. Exactly what the old metric would have hidden.
The classic model's validation went up at the end of the duel. "It started overfitting" is the automatic reading — and it was impossible: 700 steps × 8,192 tokens = 5.7 M tokens on a corpus of 60.3 M. 0.09 epoch. No token was seen twice.
The lesson is not about overfitting. It is that the most familiar explanation arrives first, and sometimes arithmetic dismisses it in ten seconds.
For three consecutive status reports, I compared the run with the compass against mini-moderno-duelo — the model without the regularizer. The correct control was mini-moderno-reg1, which differs only by the compass.
| Step | vs no-reg | vs reg1 (correct) |
|---|---|---|
| 99 | +0.04% | −1.27% |
| 149 | +0.85% | +1.06% |
| 199 | −0.30% | −0.69% |
No final conclusion changed, but three intermediate reports were wrong. Building the table with the two columns side by side made the mistake obvious — which it was not, looking at one number at a time.
Three items in the catalog are not about method: they are engineering. What they share is that none of them fails — the program runs, returns numbers, and only the card knows something is wrong.
| Problem | Effect on the card |
|---|---|
| The training script never moves the model to the GPU | the card sits idle and training runs on CPU, with no error at all |
| Document-by-document tokenization on a large corpus | preprocessing dominates the time before the first step |
| Checkpoints saved with CUDA tensors | weights that will not reload on a machine without a GPU |
All three were found by the same gesture: reading the code that was about to run, before running it. None would have shown up in a local test — on CPU everything works, just slowly.
The central idea — a latent predictor that also serves as a rudder during generation — still stands. What these pages show is that it went through a method built precisely to find its weak points.
If the shallow compass delivers little, that is written down. If the deep one delivers, the number will come with a floor, a control and a declared cost. That is what makes the claim worthy of credit.
With the method declared, chapter 07 does the arithmetic that decides the next step:
what it really costs to reach GPT-3 scale — and why the limit is not the GPU.