Attention:
the heart of the Transformer

1 The problem attention solves

Read this sentence and answer quickly: "The trophy didn't fit in the suitcase because it was too big."

"It" is the trophy. You know that instantly. Now change one word: "…because it was too small." Now "it" is the suitcase. The pronoun's referent changed because of an adjective that sits seven words later.

To process that, the model needs the representation of "it" to be able to fetch information from any other position in the sentence, choosing dynamically where to look. Not at a fixed position, not at the previous word — anywhere, depending on the content.

Why the old solution (RNN) was not enough

Until 2017, this was done with recurrent networks: read word by word, carrying a "memory state" that is updated at each step. Two fatal weaknesses:

• The memory bottleneck. The whole sentence has to be squeezed into a single state vector. Distant information dilutes and vanishes.
• It is sequential by nature. To compute word 100, you must have computed word 99. It cannot be parallelized. On a GPU with thousands of cores, that is a criminal waste — and it is what limited the scale of models for a decade.

Attention solves both at once: each position looks directly at all the others, and all positions are computed at the same time.

2 The central analogy: querying a database

Forget matrices for a minute. Think of a search system. It has three roles:

An ordinary database does hard lookup: the query matches exactly one key, and returns that value. Attention does soft lookup: the query is compared against all the keys, each one receives a compatibility score, the scores become percentages that add up to 100%, and the result is a weighted average of all the values.

3 Where Q, K and V come from

This is the point that causes the most confusion, so let's go slowly. Q, K and V are not separate things that exist in the text. They are three different projections of the same input vector.

Each token arrives with its 768-dimensional vector h (what we built in chapter 02). Then the model multiplies it by three distinct learned matrices:

Why three? Because the roles are asymmetric. What a token wants to look for is not the same thing it wants to announce about itself, and neither of those is the information it wants to deliver when it gets chosen.

• W_Q learns: "given my situation, what am I missing?"
• W_K learns: "how do I announce myself to whoever is searching?"
• W_V learns: "what do I actually contribute when I am chosen?"

If we used the same vector for query and key, attention would be forced to be symmetric — if A looks a lot at B, B would have to look a lot at A. But language is not like that: a pronoun searches for its antecedent with urgency, and the noun has no reciprocal interest in the pronoun.

4 The formula, symbol by symbol

This is the Transformer equation. I will write it and then take it apart piece by piece.

Piece 1 — QK^⊤: the compatibility scores

Multiplying Q (n×64) by K^⊤ (64×n) produces an n×n matrix. Position (i,j) of that matrix is the dot product between the query of token i and the key of token j:

This is where the quadratic cost is born. That matrix has n² entries. Doubling the length of the text quadruples the work. With n=1024 that is more than a million dot products — per head, per layer.

Piece 2 — dividing by √d_k: the reason behind the divisor

Why divide by the square root of the dimension? It is not aesthetics — it is a statistical necessity.

Suppose q and k have approximately independent components, with mean 0 and variance 1. The dot product adds up d_k of those products. By the sum of variances of independent variables:

And why does that matter so much? Because the next step is the softmax, which involves e^s. With scores on a ±8 scale, a score of 8 against one of 0 gives a ratio of e⁸ ≈ 2981 to 1. The softmax saturates: it becomes almost a one-hot, all the weight on a single token.

A saturated softmax has practically zero gradient. The model stops learning. The division by √d_k exists to keep the softmax in its useful range, where the gradient flows. Without it, the Transformer simply does not train.

Piece 3 — softmax: scores become percentages

The resulting n×n matrix A is the attention map — the object you will visualize in the lab. Each row is a probability distribution: where that token is looking.

Piece 4 — multiplying by V: the mixing

5 The dimensions, step by step

One attention head of GPT-2 small, sentence of 5 tokens:

h        (5, 768)     input (chapter 02)
  |
  |-- x W_Q (768,64) --> Q  (5, 64)
  |-- x W_K (768,64) --> K  (5, 64)
  |-- x W_V (768,64) --> V  (5, 64)

Q @ K.T           -> (5, 5)    raw scores
  / sqrt(64)      -> (5, 5)    scaled
  softmax(dim=-1) -> (5, 5)    <-- THE ATTENTION MAP (rows sum to 1)
  @ V             -> (5, 64)   head output

Note: d_k = 64 = 768 / 12 heads.  Each head works in a narrow
subspace. In chapter 04 the 12 are joined back into 768.

6 Implementation from scratch — 8 lines

import torch, math
import torch.nn.functional as F

def atencao(Q, K, V):
    """Q,K,V: (n, d_k). Returns (n, d_k) and the map (n, n)."""
    d_k = Q.shape[-1]

    # 1) every query against every key -> (n, n)
    scores = Q @ K.transpose(-2, -1)

    # 2) stabilize the scale so the softmax does not saturate
    scores = scores / math.sqrt(d_k)

    # 3) each row becomes a distribution that sums to 1
    A = F.softmax(scores, dim=-1)

    # 4) weighted average of the values
    return A @ V, A

Those are the four lines. Everything most impressive that GPT-2, RoBERTa, GPT-4 and Claude do is this operation, repeated with many heads, many layers and lots of data. The core idea fits in four lines.

7 In the lab

.venv\Scripts\python.exe lab\03_attention.py

1. Runs the implementation above with small vectors and shows the matrix A in numbers, so you can confirm that each row sums to 1.000.
2. Demonstrates saturation: computes the same attention with and without the division by √d_k, and prints the entropy of the distributions. You will see the softmax collapse.
3. Extracts GPT-2's real attention (with output_attentions=True) for the trophy sentence, and draws the map in ASCII with intensity.
4. Compares the two versions of the sentence — and here you will get two results, one of them uncomfortable. Read the box below before running.

One attention head can only look for one kind of relation at a time. And, as it stands, it lets every token see the future.
Chapter 04 solves both problems: multi-head and the causal mask.