Rafhael Oliveira · IA

Course glossary: what token, layer and model mean in AI, and in the rest of IT

36 terms AI borrowed from computing, each with the meaning used in the course, the meanings it already had in other fields, and where it appears.

1 min read· chapter: 13 min

Landed here from LinkedIn? This is the reference page Glossary of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.

"Token" already had other owners before AI: the compiler, the login and the blockchain. A reader pointed out a flaw in the course: I used token, layer, model and mask as if artificial intelligence owned those words. It does not. A token is the unit a compiler splits off when reading code, the credential a server issues after login, and the asset in a blockchain contract. A layer is a level of the OSI model. A model is a database's data model. Anyone coming from networking, databases or security reads these words with another meaning, and is right to. So the course gained a glossary: 36 terms, each with its meaning in AI, its meanings in the rest of IT and the chapters where it appears, with the sources for each field.

The three points

  1. A token was a compiler's lexical unit, a login credential and a blockchain contract's asset before it was a piece of text in a language model. All four meanings still hold.
  2. The trickiest cases are the ones that change meaning without changing field: layer, mask, cache, epoch, normalization and kernel mean one thing in networking, databases or operating systems and another inside a Transformer.
  3. Each entry lists the chapters where the term appears, and the definitions follow each discipline's reference: RFCs, the ISO standard for the OSI model, Codd for databases and Shannon for entropy.

From the chapter, the beginning

The words this course uses, with the meaning they have in machine learning — and the meaning, often quite different, that the same words already had in the rest of IT.

The field of language models borrowed a large part of computing's vocabulary. Token was already the unit a compiler splits off when it reads source code, the credential a server issues after login, and the asset recorded in a blockchain contract. Layer was already a level of the OSI model. Model was already a database's data model and the "M" in the MVC pattern. None of those meanings stopped being valid.

This course uses these words in their machine-learning sense, and that sense is neither the only one nor the oldest. Someone coming from networking, databases, security or software development reads "token", "mask" or "cache" with another meaning in mind — and is right to. This glossary exists so that reading does not depend on guessing which meaning is in use.

In this course is the meaning used in the chapters, in machine learning. In other areas of IT are the most common meanings of the same word outside that field, with the discipline in parentheses. Chapters are the chapters where the term appears. When the meaning is the same in both columns, the entry says so.

Open the Glossary →Ler em portuguêsThe PDFs by e-mail →
What the chapter covers · 5 sections · 13 min
  1. Why a glossary
  2. Text and representation
  3. Model architecture
  4. Training and evaluation
  5. Generation and execution
All chapters of the course · GPT-1 and GPT-2, taken apart
  1. 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
  2. Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
  3. 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
  4. 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
  5. 03 · Attention: the single operation that replaced twenty years of recurrent networks
  6. 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
  7. 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
  8. 06 · What scale really buys: from 124 million to 1.5 billion, measured
  9. 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
  10. 08 · Perplexity: the one number that says what the model learned, without anecdotes
  11. 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
  12. 10 · RoBERTa: when the advance is training better what already existed
  13. 11 · Encoder vs decoder: one line of code and every consequence that descends from it
  14. 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
  15. 13 · Reading GPT-2's original code: 174 lines against everything you learned
  16. 14 · Serving models: from function call to HTTP server, and what production does on top
  17. 15 · The lab guide: every command, every script, and everything that was measured
  18. 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2

After this: the Antivision experiment

Rafhael Oliveira
Rafhael Oliveira

Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.

Follow on LinkedIn →