GPT-2 · RoBERTa Lab · Part I — The foundation
02

Embeddings:
where meaning is born

From a meaningless integer to a vector of 768 numbers that knows "king" and "queen" are related.

Chapter 02 of 15 Prerequisite: chapter 01

1 The problem: integers have no meaning

We left chapter 01 with a list of integers. Say gato (cat) = 4899 and cachorro (dog) = 5257. These numbers are locker labels, not measurements. Locker 5257 is not "bigger" or "hotter" than 4899. If the vocabulary were shuffled, cat would be 12 and dog would be 40,001, and nothing would change.

But a neural network only knows how to do arithmetic. If it receives 4899 and 5257, it will inevitably compute differences, averages, products — and it will extract relations that do not exist. Feeding raw IDs into a network is a serious conceptual error.

The first solution that usually comes up, and why it does not work: one-hot

The classic way of saying "this is category 4899, and categories have no order" is the one-hot vector: a vector of 50,257 positions, all zero, except for a 1 at position 4899.

That solves the false-order problem. But it creates two new ones:

2 The idea: give words coordinates

What if, instead of 50,257 positions with one lonely 1, each word had 768 continuous numbers? Not chosen by us — learned by the model during training, adjusted at every mistake it makes.

Those 768 numbers are the coordinates of the word in a 768-dimensional space. And here is the central point: if training goes well, words that appear in similar contexts end up with similar coordinates. They cluster. The space acquires semantic geography.

Analogy

Think of a map of Brazil. Each city is a pair of numbers (latitude, longitude). Those two numbers are not arbitrary: cities with similar numbers really are close, and you can do useful arithmetic with them ("which city is 200 km to the north?").

An embedding is the same thing, with 768 coordinates instead of 2. And the "directions" of that map are not north-south, but things the model discovered on its own: maybe one direction is "formality", another "singular↔plural", another "concrete↔abstract". Nobody programmed those axes — they emerged because they helped predict the next word.

The classic demonstration

In well-trained embedding spaces, semantic relations become constant vector displacements:

v(king) − v(man) + v(woman) ≈ v(queen) The vector going from "man" to "woman" is approximately the same as the one going from "king" to "queen". The gender relation became a fixed direction in the space. This was not programmed: it was a consequence of predicting text.

3 In practice: the matrix is a lookup table

Formally, the embedding layer is a matrix:

We ∈ ℝ|V| × d  =  ℝ50257 × 768 |V| = vocabulary size (one row per possible token). d = dmodel, the width of the model. In GPT-2 small, d = 768. Total: 38.6 million parameters in this matrix alone — 31% of the entire model.

Mathematically, fetching the embedding of token i is multiplying the one-hot by the matrix:

ei = onehot(i)⊤ We But multiplying a vector of 50,257 zeros by a matrix is throwing 50,257 multiplications in the trash to pick one row.

In the real implementation, nobody does that multiplication. You do direct indexing: W_e[4899]. That is why the layer is called a lookup table. The one-hot exists only in the mathematical explanation; in the code it is never built.

Remember this name

In GPT-2 this matrix is called wte — word token embedding. You will see it in the weight names: transformer.wte.weight. And you will meet it again at the end of the model, in a surprising way (section 6).

4 The second embedding: position

There is a hole in what we have built so far, and it is a serious one. Attention, which you will see in chapter 03, is a set operation: it looks at all tokens at the same time, with no notion of order whatsoever. To it, "the dog bit the man" and "the man bit the dog" are identical — same tokens, same set.

That is unacceptable for language. Order is meaning. So we need to inject position back in, explicitly.

How GPT-2 solves it: learn the position too

The original Transformer paper (2017) used sines and cosines of different frequencies — a fixed, elegant formula that allows extrapolating to lengths never seen. GPT-2 dropped that and made the simplest possible choice: a second lookup table, learned from scratch.

Wp ∈ ℝnctx × d  =  ℝ1024 × 768 nctx = 1024 is GPT-2's maximum context length. One row per possible position: row 0 is "the vector for being the first token", row 1 is "the vector for being the second", and so on.

In GPT-2 this matrix is called wpe — word position embedding.

The hard consequence of "learned"

Since wpe has exactly 1024 rows and each one was learned individually, GPT-2 is physically incapable of processing more than 1024 tokens. It is not a conservative setting that can be raised — position 1025 simply does not exist in the matrix. Trying to use it is an index error.

This contrasts with the original sinusoidal Transformer (which extrapolates) and with modern techniques such as RoPE (used in Llama, Qwen, Mistral), which allow the context to be stretched. It is a 2019 design limitation that you will bump into in practice.

5 Putting it together: the model's real input

The two embeddings are simply added, element by element:

h0 = We[tokens] + Wp[0, 1, …, n−1] h0 is "hidden state zero" — the input to the first Transformer block. Shape: (n tokens × 768). Each row is a token that already knows what it is and where it is.

"Add? Shouldn't it concatenate?"

That is the right question, and it is a common one. Concatenating (768 of content + 768 of position = 1536) looks cleaner, but it would double the width of the model and the cost of everything after it.

Addition works because 768 dimensions is a lot of room. In high dimension, two random vectors are almost always practically orthogonal — they occupy different "directions" and do not get in each other's way. The model learns to use certain directions for content and others for position, and the following layers manage to separate back what was added. It is counterintuitive, but it is a real property of high-dimensional spaces.

A concrete example of sizes

Input: "O Brasil é gigante"   ->  5 tokens

wte[ids]        ->  (5, 768)     WHAT each token is
wpe[0..4]       ->  (5, 768)     WHERE each token is
                    ---------
h_0 = sum       ->  (5, 768)     input to block 1

This (5, 768) goes through the 12 blocks WITHOUT CHANGING SHAPE.
(5, 768) comes out of block 12. The shape is an invariant of the Transformer.

6 The final knot: weight tying

Here is a beautiful engineering detail, one that many explanations of GPT-2 forget.

At the end of the model, you need to convert the 768-dimensional vector of the last position back into a score for each of the 50,257 possible tokens. That calls for an output matrix Wout ∈ ℝ768 × 50257 — another 38.6 million parameters.

GPT-2 does not create that matrix. It reuses the embedding matrix, transposed:

logits = hfinal We⊤ The same We that converted token → vector at the input now converts vector → scores at the output. Zero new parameters.

This is called weight tying, and it has two justifications:

7 The code

# what lives inside wte and wpe
import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
m   = AutoModel.from_pretrained("openai-community/gpt2")

wte = m.wte.weight   # (50257, 768)  content
wpe = m.wpe.weight   # (1024,  768)  position

# 1) a token's embedding is just one row of the table
id_gato = tok.encode(" gato")[0]
v = wte[id_gato]                    # vector of 768 numbers

# 2) cosine similarity = "how similar are two words?"
def prox(a, b):
    va, vb = wte[tok.encode(a)[0]], wte[tok.encode(b)[0]]
    return torch.cosine_similarity(va, vb, dim=0).item()

print(f"king  x queen  {prox(' king',' queen'):+.3f}")   # high
print(f"king  x banana {prox(' king',' banana'):+.3f}")  # low

# 3) proof of weight tying: the two matrices are the SAME object
lm = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
print(lm.lm_head.weight.data_ptr() == lm.transformer.wte.weight.data_ptr())
#    True  -> same memory address. Not a copy, the same tensor.

8 In the lab

.venv\Scripts\python.exe lab\02_embeddings.py
  1. Shows the raw vector of a token (the first 12 of the 768 numbers). It is there so you can see there is nothing mystical: they are floats.
  2. Ranking of semantic neighbors: given a word, which are the 10 closest tokens by cosine. Notice that many neighbors are variations of the same word (capitalized, plural, with a space) — the space is dominated by form as much as by meaning.
  3. The king−man+woman analogy actually running. And it works: " queen" comes out at +0.709 and " princess" right behind. But the script also shows the caveat that the pretty demos usually hide — " king" shows up in first place, because we started from its vector and moved it only a little. Almost every famous demo of this analogy removes the input words from the ranking before showing the result.
  4. Compares the matrix sizes across the four GPT-2s, showing that wte is the largest single piece in all of them.
  5. Verifies weight tying by comparing memory addresses.
The real limit of this layer: it is static

wte gives one fixed row per token, before any context. That is: "banco" as in a park bench and "banco" as in a bank (the Portuguese word means both) receive exactly the same vector. In this layer, the ambiguity is unresolvable by construction.

And GPT-2 does not need it to resolve it. It has 12 attention blocks after it precisely to build contextual meaning — the "banco" will specialize according to its neighbors. wte is a rough starting point; the real semantic map lives in the middle layers. Remember this: it is the difference between static embeddings (Word2Vec, GloVe, wte) and contextual ones (the output of any Transformer layer), and it is the reason BERT was a revolution in 2018.

Exercise 2 — predict before running

  1. Compute the cosine between " cat" and " cats", and between " cat" and " dog". Which pair do you expect to be closer — the one by form or the one by meaning? Run it and see whether your intuition was right.
  2. How much do wte + wpe take up in MB in GPT-2 XL (vocabulary 50,257, d=1600, context 1024), in fp32? Compute it by hand first: each float takes 4 bytes.
  3. wpe has 1024 rows. Take rows 0, 1, 2 and 500, and compute the cosine between them. Should neighboring positions have similar vectors? Do you think the model learned that on its own? (Run it: the answer is yes, and it is beautiful.)
  4. Hard: if weight tying makes the same matrix serve both input and output, the gradient reaching it comes from two paths. Does that help or hurt training? Argue both sides.
References

The primary sources for the concepts in this chapter. The numbers in the text were measured in the lab; the ideas come from here.

Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.

  1. Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. ICLR 2013, workshop. arXiv:1301.3781
  2. Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762
  3. Press, O. and Wolf, L. (2017). Using the Output Embedding to Improve Language Models. EACL 2017. arXiv:1608.05859

Now every token is a vector that knows what it is and where it is — but it is still alone, not talking to its neighbors.
In chapter 03 they finally talk. It is the heart of the Transformer.