Attention: the single operation that replaced twenty years of recurrent networks
Query, Key and Value, where they come from, what they mean, and the 8-line implementation.
Landed here from LinkedIn? This is chapter 03 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.
"The trophy didn't fit in the suitcase because it was too big." What is 'it'? You answered instantly: the trophy. Now change one word: '...because it was too small'. Now 'it' is the suitcase. The pronoun's referent changed because of an adjective seven words later. Twenty years of recurrent networks did not handle that well. A single operation did, and it is a database lookup. Except, measured on GPT-2, there is a catch: the attention of 'it' is identical in both sentences. Bit for bit. Difference exactly 0.00.
The three points
- The central analogy: each token makes a query (Q) against the keys (K) of all the others and brings back the values (V). It is a differentiable database.
- The formula has four symbols and each has a reason. The worst explained is the division by the square root of the dimension.
- In the lab, the attention of 'it' comes out identical in both trophy sentences. In a causal model, the past cannot be affected by the future. A RoBERTa would differ.
From the chapter, the beginning
Query, Key and Value — where they come from, what they mean, and why this single operation replaced twenty years of recurrent networks.
Read this sentence and answer quickly: "The trophy didn't fit in the suitcase because it was too big."
To process that, the model needs the representation of "it" to be able to fetch information from any other position in the sentence, choosing dynamically where to look. Not at a fixed position, not at the previous word — anywhere, depending on the content.
What the chapter covers · 7 sections · 10 min
- The problem attention solves
- The central analogy: querying a database
- Where Q, K and V come from
- The formula, symbol by symbol
- The dimensions, step by step
- Implementation from scratch — 8 lines
- In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
- 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
- Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
- 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
- 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
- 03 · Attention: the single operation that replaced twenty years of recurrent networks
- 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
- 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
- 06 · What scale really buys: from 124 million to 1.5 billion, measured
- 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
- 08 · Perplexity: the one number that says what the model learned, without anecdotes
- 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
- 10 · RoBERTa: when the advance is training better what already existed
- 11 · Encoder vs decoder: one line of code and every consequence that descends from it
- 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
- 13 · Reading GPT-2's original code: 174 lines against everything you learned
- 14 · Serving models: from function call to HTTP server, and what production does on top
- 15 · The lab guide: every command, every script, and everything that was measured
- 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →