Consulted
memory

1 What the summed memory does not do

The memory of chapter 11 is one vector m per sequence, updated at every block that leaves the window: m ← m + f([m ; s_block]). It persists — prediction with the memory of the right text gains +0.019 over the memory of another text, even a whole window later. But the same table showed the limit: "long" (A+B) errs 0.181, worse than "recent" (B only, 0.173) and than "old only" (A only, 0.174). A vector of fixed capacity does not accumulate: each new block overwrites part of the previous one. The more past, the less of each piece.

LeCun describes another piece. The talk: "you may have to combine this with the content of a memory to get a complete idea of the state of the world". The interview: systems with "persistent memory, of the type that we have". And the architecture slide has the memory as a module consulted by the world model, not as a running sum. Chapter 11 left this written as an open question: "the consulted memory, not just summed, requires attention over a bank of abstract states, and is another piece". This is the piece.

2 The piece: bank + attention

Every block of 8 tokens that leaves the window is abstracted by the same level-2 encoder (abstrair()) and stored, not summed. The bank is the set S = {s₁, …, s_n} of those states. The world model consults the bank with a key projected from the current state, and reads the result by attention:

q = W_q z_t / √d   r_t = softmax(q · S^T) S   Pred(z_t, a_t[, ζ][, m], r_t) → z_t+1

Three decisions, all inherited from what was already measured. The encoder receives no gradient through the bank (only W_q and the world model learn to consult it; the same rule as the summed memory, for the same reason: otherwise the encoder hides the target in the bank). Training uses the double windows of chapter 11: the training bank is the blocks of the contiguous past, and the attention is exact over them (at most 32 states). And a bank-dropout of 0.5 is built in — the lesson of chapter 13: half the batches with r = 0, so the predictor works when the bank is empty, which is the case for every short prompt.

The measure is chapter 11's, transposed: bank_gain = MSE(bank of another text) − MSE(right bank), with the MSE of the empty bank alongside. Zero means the attention finds nothing useful in the past.

3 Why a vector index, and why turbovec

In training the bank fits in one matrix multiplication. On the bench it does not: the bank grows with every block generated or read, session after session, and the interview's promise is that it is not erased. One million blocks (eight million tokens read) are 256 million floats — a gigabyte in fp32, and an exact attention over a million states at every token. What the attention actually needs are the k largest logits q · s_j; the rest of the softmax is negligible. And "the k largest inner products with a query" is exactly what a vector index returns.

turbovec is a Rust index with Python bindings implementing TurboQuant (Google Research, 2025): each vector is normalized, goes through a fixed random rotation (which makes every coordinate approximately Beta-distributed, with the same distribution in all dimensions), is quantized per coordinate with a 4-bit Lloyd-Max codebook (16 levels) and bit-packed — 256 dimensions become 128 bytes, eight times less than fp32 — and one scalar per vector corrects the bias of the estimated inner product. Search runs in SIMD over the codes, with no training step and no parameter tuning: the bank ingests online, one block at a time, which is the bench's regime. Saving is incremental (sync writes only what changed since the last time), which makes persistence cost milliseconds per generation. What it does not return are the vectors — only ids and scores — so the fp32 states live in a file alongside, and the attention is done over the k actually retrieved.

The simplest alternative would be to keep everything in RAM and search exactly; it stays available as a fallback (same API, without turbovec installed) and is the reference for measuring what quantization costs in recall — the question section 5 leaves open.

4 The round

Round wm1n2bco-sz1, on the same budget as the others: hierarchy with the raw mean (the honest abstraction of chapter 11), SigReg-z, bank with dropout 0.5, without the summed memory (to measure the piece alone). Three questions: does the bank carry the past (gain > 0)? is the whole past worth more than each half (what the vector could not do)? and does the rudder govern on this map, with the bank empty — which is the condition of chapter 12?

The bank solves the dilution that was the summed memory's defect. In chapter 11, "long" (A+B) erred 0.181, worse than "recent" (B only, 0.173): the more past, the less of each piece, because a fixed-capacity vector overwrites itself. In the consulted bank, "long" errs 0.1736 — exactly equal to "recent", and slightly better than "old only" (0.1738). Storing instead of summing removes dilution by construction: the old block is still there, intact, and attention picks. It is what the slide's architecture predicted and the summation did not deliver.

And the bank carries little. Its gain over another text's memory is +0.0036, against chapter 11's +0.014 for the summed memory and +0.002 for the summed one with dropout. The cause is the same as in both earlier cases, and it is in the recipe: bank-dropout of 0.5 teaches the predictor to work without a bank, and it learns not to depend on one. The difference is that here the dropout was mandatory — every short prompt starts with an empty bank — so the trade-off is not a calibration error but the shape of the problem. The round with lower dropout is the missing test.

The rudder does not govern on this map (0.272 with head and real state against 0.360 free), and the cost to the generator is the family's highest, +9.73%. Summing up: consulted memory delivers what it promised on persistence — it does not dilute — and delivers nothing else. Under chapter 13 §12.3's noise ruler, +9.73% is indistinguishable from the same map's +9.55% without a bank: the bank is free and carries almost nothing. It stays on the bench as infrastructure, with the quantised on-disk index ready for a large bank, and the dropout question open.

5 On the bench

• Every model trained with --world-bank gets a bank in runs/<round>/banco/ (banco.tvim + states in .npy), loaded when the model is loaded and saved at the end of every generation. Every block that leaves the window — from the prompt or from what was generated — enters the bank.
• compass parameters: banco (true, false, or "limpar" to clear), banco_gravar (false = consult only) and banco_k (top-k, default 8). The trajectory returns banco_estados.
• The experiment (model/memory.py) prints the bank's table next to the summed memory's, when the model has both.