The bench

1 What the bench is

A local page, no framework, no build and no CDN — it works offline. Five tabs over the same FastAPI API:

The Playground selector has two groups: OpenAI (only gpt2-xl) and Ours (all checkpoints). The other four HuggingFace models remain downloaded and keep serving the labs — they just left the selector, so it does not become a list of things nobody picks.

2 Where the time of a request goes

Before optimizing anything, we measured. gpt2, 23-token prompt, 32 tokens generated, 8 threads:

What the KV cache pays

1.62× — less than the "10×" one usually hears, because with 32 tokens the context is still short and the gain is quadratic in length. The cache is only possible because the mask is causal: the past never changes, so its K and V can be stored. In an encoder like RoBERTa that does not exist.

The throughput our server throws away

Each generated token reads all of the model's weights from memory. With batch 1, that read serves one request; with batch 16, sixteen — for the same cost. It is nearly free throughput, and it is the reason continuous batching exists in TGI and vLLM.

Our server handles one at a time. For a study bench that is right; for production, it is the first bottleneck to attack. It is documented as a conscious limitation, not as an oversight.

The GPT-2 wall, measured

contexto  512  → 29,6 ms/token
contexto 1000  → 34,6 ms/token
contexto 1024  → PAREDE: posição 1024 não existe na wpe (ela tem 1024 linhas: 0 a 1023)

The error that broke the measurement script on its first run is literally GPT-2's architectural limitation — and it is exactly what RoPE eliminates (chapter 02).

3 The cache with a budget

The first version kept every loaded model in memory, forever. The seven models in float32 add up to ~13 GB — above the virtual memory of an ordinary development machine — and the operating system kills the process once it fills up:

OSError: the paging file is too small for this
operation to complete. (os error 1455)

There was a second bug alongside it, more subtle: the cache key included whether the model had been loaded with an attention map or not. Opening the Playground and then the X-Ray on the same model put two copies of it in memory. Nobody noticed — only the RAM disappearing.

The fix was a cache with a budget: an LRU with a ceiling (LAB_CACHE_GB) that evicts the least-used model when it is exceeded, with GET /info showing the state. Tested:

gpt2        → 0.46 GB
+ medium    → 1.79 GB
+ large     → 4.67 GB
+ gpt2-xl   → 5.80 GB   ← evicted the three smallest
+ det-large → 1.32 GB   ← evicted the xl

And attn_implementation="eager" became always-on — one entry per model, and it is what the X-Ray needs anyway.

4 The Laboratory tab

The project's nine scripts became forms, generated from a catalog on the server. The log arrives line by line via Server-Sent Events while it runs; closing the tab does not kill the process, and coming back shows the log from the beginning.

Security: no "run this command"

An open port that executes arbitrary commands is an open port that executes arbitrary commands, even on localhost. The design is a closed list: each script declares its arguments with type, minimum and maximum, and the argv is assembled by code — never concatenated into a shell string.

montar('treinar', {'rodada': 'a; rm -rf /'})
  -> rodada: use apenas letras, numeros, - _ . e virgula

montar('treinar', {'passos': 999999})
  -> passos: maximo 200000

5 The Models tab

One card per training run saved in model/runs/, with the curve drawn in pure SVG — training loss in gray, validation in green, anisotropy in dashed purple. The badges are read from the folder name: moderno, GPT-2 fiel, λ=1.

Below, a field that measures bits/character across all models at once and highlights the best. It is the honest metric across different tokenizers — perplexity is measured per token, and comparing PPL across vocabularies means nothing.

6 The API

The compass parameter on /generate is the extension chapter 05 describes: {"alvo": "…", "alpha": 0, "beta": 8}.

Serving our models is different code

The HuggingFace models come with generate(), cache, streamer and an entire ecosystem. Ours is a 400-line nn.Module. Instead of pretending they are the same, the difference lives in server/ours.py, in one place only, and the rest of the server deals with a single interface.

Generation decodes incrementally and sends only what grew — byte-level BPE may need two tokens to form one character, and decoding each token in isolation would put garbage on the screen.

Result: our mini generates at 127 tok/s against ~7 for gpt2. Two hundred times smaller, eighteen times faster.

Chapter 09 shows the whole architecture, aligned with the world model.
Chapters 04 and 05 — the lineage of collapse and the rudder — wait for the experiments to close: writing them earlier would be making up numbers.