Memory and horizon: two claims from LeCun that became experiments, and both hold for text

LeCun says: each level of abstraction predicts further than the one below. We measured. It holds for text. Two of his claims became experiments. The first: combine perception with the content of a memory. An ordinary language model does not have that, it has the window. We added a memory vector that persists, and the memory of the right text predicted better than another text's memory even a whole window later. The second: the abstract level, which never saw the individual tokens, predicts the next block in one step with four times the gain of the token level rolled eight times. At the honest scale of a 7.4-million model.

The three points

1. Memory: the right text's predicts better than another text's (0.173 against 0.193), and the one from a window ago is worth as much as the recent one. Persistence, not recency.
2. The limit: the whole past is worth less than each half. Summed memory dilutes. It is what motivates chapter 14.
3. Horizon: level 2 in one step gains +0.078 over the floor; level 1 rolled eight times gains +0.020. Abstraction predicts further, four times.

From the chapter, the beginning

Two claims from LeCun that became experiments: "combine perception with the contents of a memory" and "each level of abstraction predicts further than the one below". Both were measured, and both hold for text — at the honest scale of a 7.4 M model.

LeCun lists persistent memory, in both talks, as a requirement of an intelligent system, and the talk says where it enters: perception gives only the current state; the full state is perception combined with the contents of a memory. An ordinary language model has no such thing — it has the context window, and what leaves it ceases to exist.