All posts

Not a course. An experiment of proof and continuous improvement: every piece is measured against a control, what fails is recorded with the same prominence as what works, and the next run starts from what the previous one measured. The chapters are the diary of that experiment, in the order it happened.

1. 00
   Antivision: an ELLM, a language model that rehearses in latent space before speakingAn LLM picks the next token and finds out afterwards where it ended up. We built one that imagines first: an ELLM.highlightclosedupdated 12 Sept
2. 01
   The custom tokenizer: why 16 thousand tokens suit this corpus better than 50 thousand, and what the whole table showsA 7-million-parameter model used fewer bits than the 124-million GPT-2 in Portuguese. Not because it is better.closedupdated 12 Sept
3. 02
   Two architectures, one file: the fair duel between 2019 and what came afterA new piece is presented under the conditions of whoever built it. We redid the comparison under ours: same data, same seed, same budget.closedupdated 12 Sept
4. 03
   The geometry of the embedding: training starts with a perfect space and ruins itWe tried semantic search with RoBERTa. Every similarity came out above 0.99. Sentences on opposite subjects looked identical.closedupdated 12 Sept
5. 04
   The lineage of collapse: five answers from computer vision, each measured on textA system trained to predict in latent space has a trivial solution: ignore the input and always return the same vector. Perfect and useless prediction.closedupdated 12 Sept
6. 05
   The rudder and the world model: three versions of the same question, each measured against the lastThe question is a single one: does the model know where it is going before it goes?highlightclosedupdated 12 Sept
7. 06
   Method and mistakes: the rules of a legitimate comparison and the catalogue of our own errorsHalf of the best decisions in this project came from finding out we were wrong.closedupdated 12 Sept
8. 07
   Scale and budget: what it costs to reach GPT-3, and why VRAM is almost never the limitReaching the scale of GPT-3 Small costs less than eight hours of a single GPU. What is missing is not the card. It is text.closedupdated 12 Sept
9. 08
   The workbench: serve, x-ray and train from the browserWe measured where the time of a request goes. 93.3% is serial generation, token by token.closedupdated 12 Sept
10. 09
    The hierarchical architecture: the whole project aligned to the four patterns of the world modelLeCun has a slide with four lines: abandon X in favour of Y. We audited our code against each one.closedupdated 12 Sept
11. 10
    What JEPA is, according to LeCun: two talks read concept by concept, and where each one appliesThe video that started it all: Yann LeCun explaining what LLMs still lack to understand the world. We read it concept by concept and applied it to text.closedupdated 12 Sept
12. 11
    Memory and horizon: two claims from LeCun that became experiments, and both hold for textLeCun says: each level of abstraction predicts further than the one below. We measured. It holds for text.closedupdated 12 Sept
13. 12
    Learned costs and inference modes: the three searches and the two sources of energy, measuredOn the first reading, no inference mode governed. Free generation had more heading than any planner.highlightclosedupdated 12 Sept
14. 13
    The corrections from LeCun's talks: failure, passage, flag, measurementA queue of 22 runs left four measured failures. We took each back to LeCun's text with a single question.highlightclosedupdated 12 Sept
15. 14
    Consulted memory: a bank of abstract states read by attentionSummed memory persists, but dilutes: the whole past is worth less than each half alone.closedupdated 12 Sept
16. 15
    What it costs and what it delivers: us, GPT-1 and GPT-2 on the only ruler valid across tokenizers"Our model has perplexity 31, GPT-2 has 29, so GPT-2 is better." That means nothing.closedupdated 12 Sept
17. 16
    The minimum was off the list: the day I measured my own search by energyFor weeks I said my system picked the next word by "searching for the lowest energy". I went and measured it. In 96 tests, it never found the lowest energy once.highlightclosedupdated 12 Sept
18. 17
    ELLM: the right name for a language model that predicts in latent space and decides by energyI called my model an LLM for months. The name was wrong — and not because of its size.highlightclosed
19. 18
    Searching better does not fix the energy: three searches, the same textI fixed my model's search so it finds the true lowest energy, among 16,392 options. The text did not get better.recenthighlightclosed
20. 19
    The inversion that did not repeat: the world model's cost appears once the ruler stops swingingI wrote in a chapter that my world model made the generator 3.38% better. With more seeds and a fixed ruler, it costs.recenthighlightclosed
21. 20
    What changed on 17/09: two new chapters and two correctionsIn one day the experiment gained two chapters and corrected two claims I had already published.recentclosed