Embeddings: from a meaningless integer to a vector that knows king and queen are related

To the model, cat is 4899 and dog is 5257. Feeding that into a neural network is a serious conceptual error. Those numbers are locker labels, not measurements. Locker 5257 is not bigger or warmer than 4899. But a neural network only knows arithmetic: given the two integers, it will compute differences and averages and extract relationships that do not exist. The fix was to give words coordinates. 768 of them, in GPT-2 small. And it is in that space that 'king' and 'queen' become relatives.

The three points

1. The embedding matrix is a table: row 4899 is the vector for cat. It is learned along with everything else.
2. There is a second embedding, for position, because attention alone does not know word order.
3. Weight tying: the same matrix that turns a token into a vector turns a vector back into a token. It saves 38 million parameters in GPT-2 small.

From the chapter, the beginning

From a meaningless integer to a vector of 768 numbers that knows "king" and "queen" are related.

We left chapter 01 with a list of integers. Say gato (cat) = 4899 and cachorro (dog) = 5257. These numbers are locker labels, not measurements. Locker 5257 is not "bigger" or "hotter" than 4899. If the vocabulary were shuffled, cat would be 12 and dog would be 40,001, and nothing would change.

But a neural network only knows how to do arithmetic. If it receives 4899 and 5257, it will inevitably compute differences, averages, products — and it will extract relations that do not exist. Feeding raw IDs into a network is a serious conceptual error.

The classic way of saying "this is category 4899, and categories have no order" is the one-hot vector: a vector of 50,257 positions, all zero, except for a 1 at position 4899.