Consulted memory: a bank of abstract states read by attention

Summed memory persists, but dilutes: the whole past is worth less than each half alone. Chapter 11 measured it: long memory errs 0.181, recent-only memory errs 0.173. Summing the past into a vector is losing what you sum. LeCun asks for something else: a consulted memory, which survives the window and the session. Here it becomes a bank of abstract states read by attention, with a vector index quantised to 4 bits: 128 bytes per state instead of 1,024, a million blocks in 128 MB. The run that measures whether attention chooses from the whole past instead of diluting it is in the queue.

The three points

1. What the sum does not do: the whole past is worth less than each half. Attention over a bank chooses instead of diluting.
2. Why a vector index: a million blocks fit in 128 MB at 4 bits, with SIMD search and incremental persistence. Attention uses only the k retrieved, in fp32.
3. How to read the run: the bank passes if 'long' beats 'recent' and 'empty' stays neutral. If the gain is zero, the next step is a longer past, not another piece.

From the chapter, the beginning

Chapter 11 measured that a memory vector persists, but dilutes: the whole past is worth less than each half. LeCun asks for something else — a memory that is consulted, and survives the window and the session. Here it becomes a bank of abstract states read by attention, with a quantized vector index (turbovec) keeping the bank on disk.

The memory of chapter 11 is one vector m per sequence, updated at every block that leaves the window: m ← m + f([m ; sblock]). It persists — prediction with the memory of the right text gains +0.019 over the memory of another text, even a whole window later. But the same table showed the limit: "long" (A+B) errs 0.181, worse than "recent" (B only, 0.173) and than "old only" (A only, 0.174). A vector of fixed capacity does not accumulate: each new block overwrites part of the previous one. The more past, the less of each piece.

LeCun describes another piece. The talk: "you may have to combine this with the content of a memory to get a complete idea of the state of the world". The interview: systems with "persistent memory, of the type that we have". And the architecture slide has the memory as a module consulted by the world model, not as a running sum. Chapter 11 left this written as an open question: "the consulted memory, not just summed, requires attention over a bank of abstract states, and is another piece". This is the piece.