The words this course uses, with the meaning they have in machine learning — and the meaning, often quite different, that the same words already had in the rest of IT.
The field of language models borrowed a large part of computing's vocabulary. Token was already the unit a compiler splits off when it reads source code, the credential a server issues after login, and the asset recorded in a blockchain contract. Layer was already a level of the OSI model. Model was already a database's data model and the "M" in the MVC pattern. None of those meanings stopped being valid.
This course uses these words in their machine-learning sense, and that sense is neither the only one nor the oldest. Someone coming from networking, databases, security or software development reads "token", "mask" or "cache" with another meaning in mind — and is right to. This glossary exists so that reading does not depend on guessing which meaning is in use.
In this course is the meaning used in the chapters, in machine learning. In other areas of IT are the most common meanings of the same word outside that field, with the discipline in parentheses. Chapters are the chapters where the term appears. When the meaning is the same in both columns, the entry says so.
| term | in this course | in other areas of IT | chapters |
|---|---|---|---|
| token | the piece of text the model handles as a single unit — it can be a whole word, part of one, or even a byte. Each piece has an ID number. house is one piece; produtividade is four | an access credential issued after authentication, such as a JWT (security); the lexical unit a compiler's scanner splits off, such as a keyword or identifier (compilers); a digital asset recorded in a contract (blockchain); a physical authentication device (security) | 01–16 |
| tokenization, tokenizer | cutting text into those pieces, before the model sees anything. The tokenizer is the program that does it, working from a fixed list of pieces, decided before training and never changed again | replacing sensitive data, such as a card number, with a meaningless surrogate value (payment security); the lexical analysis stage (compilers) | 00, 01, 04, 06, 08–11, 13, 15, 16 |
| vocabulary | the complete list of pieces the tokenizer knows — and it is closed: nothing outside it exists for the model. In GPT-2 there are 50,257 pieces | controlled vocabulary: a standardized list of terms for classifying information (ontologies, semantic web) | 00–02, 06–10, 12, 13, 16 |
| embedding | the translation of a piece of text into numbers. Each token becomes a list of numbers the model learned on its own during training, and words with similar meanings end up with similar lists. The table holding all those lists is also called an embedding | an embedded system (hardware); embedding external content in a page, such as a video (web) | 00–02, 05, 06, 11, 14–16 |
| vector | a list of numbers in order, treated as a point in space. Because a point has a position, two vectors have a distance and an angle between them — and that is how the model measures whether two things are alike | a data structure with elements in contiguous positions, the array (programming); an interrupt vector (operating systems); a vector image (computer graphics) | 01–05, 11, 12, 14, 16 |
| dimension | each slot inside a vector. If the vector is a list of 768 numbers, it has 768 dimensions; that is the size of GPT-2 small | a dimension table, describing facts in a data warehouse (dimensional modelling); the size of a screen or image in pixels (interfaces) | 00, 02–05, 11, 12, 14 |
| encoding | here, almost always positional encoding: the extra number that tells the model where in the sentence a piece sits. Without it, the dog bit the man and the man bit the dog would be identical to it. Formally, a vector added to the embedding | character encoding, such as UTF-8 (systems); an audio or video compression format, the codec (multimedia) | 00, 07, 15, 16 |
| term | in this course | in other areas of IT | chapters |
|---|---|---|---|
| model | the whole trained program: it takes tokens and returns, for every possible piece, the chance of it coming next. In practice it is two things together: the design (the architecture) and the learned numbers (the weights) | a data model, such as entity-relationship (databases); the "M" in the MVC pattern, the layer holding the data (software engineering); a software development process model (software engineering) | 00–16 |
| layer | a block of computation repeated, identically, several times — the output of one is the input of the next. GPT-2 small stacks 12 Transformer blocks, and each block holds an attention and a small network that processes each position on its own (the MLP) | a level of the OSI model or the TCP/IP stack (networking); a tier of a layered architecture, such as presentation, business and data (software engineering) | 00–06, 09, 11–15 |
| attention | the operation that lets each word look at the others in the sentence and decide which ones it depends on. In the dog I saw yesterday barked, it is what links barked to dog, not to yesterday. Technically: an average of the other tokens, weighted by what each one contains | no technical meaning outside the field; the name comes from the idea of focus and does not describe human attention | 00–11, 14–16 |
| head | two things share the name. (1) One of the attentions running in parallel inside the same layer — each learns to notice a different kind of relation. (2) The last piece of the model, which turns numbers into odds for each word: the “language head” | a disk's read/write head (hardware); HEAD, the reference to the current commit in Git (version control); the HTTP HEAD method (web) | 00, 03–06, 10–12, 14, 15 |
| mask | what stops the model from seeing what it should not. In GPT it hides the future: predicting word 5, it cannot look at word 6 — otherwise it would not be predicting, it would be copying. In BERT it is the opposite: it hides a word in the middle of the sentence and asks the model to guess it | a subnet mask (networking); a bitmask (programming); an input mask that formats a field (interfaces) | 00, 03, 04, 08, 10, 11, 13–16 |
| encoder, decoder | the two ways to build a Transformer. Encoder: every word sees the whole sentence, ideal for understanding — that is BERT. Decoder: every word sees only what came before, necessary for writing — that is GPT. The difference fits in one line of code | character or media encoder and decoder (systems, multimedia); a rotary encoder that measures position (hardware) | 00, 04, 09–16 |
| parameter | each number the model adjusted on its own during training; everything it “knows” lives in them. GPT-2 small has 124 million. What you choose before training is not a parameter, it is a hyperparameter | a value passed to a function or command (programming); a configuration option of a system (systems administration); a query parameter in a URL (web) | 00, 02, 03, 05–09, 11–13, 15, 16 |
| weight | the most common kind of parameter: the number an input gets multiplied by. When someone says “I downloaded the model's weights”, that is the file with all those trained numbers | the weight of an edge in a weighted graph (algorithms); a distribution weight in a load balancer (networking) | 00–07, 10, 12–15 |
| bias | the number that is added after the multiplication by the weights — the | algorithmic bias: an outcome that systematically favours or harms a group (data ethics) | 01, 06, 09, 15 |
| normalization | rescaling a list of numbers so they sit in a similar range, with no giant values next to tiny ones. That keeps training stable; in the Transformer it is LayerNorm, which rescales each vector to a controlled mean and spread | normal forms, which remove redundancy from tables (databases); Unicode normalization, which unifies equivalent characters (systems) | 04–07, 09, 14, 16 |
| state | the hidden state: how a piece of text is represented after it has been through the layers. Same word, but now carrying the context of the whole sentence | the state of a program or of a state machine (programming); session state (web) | 00–03, 05, 07, 16 |
| term | in this course | in other areas of IT | chapters |
|---|---|---|---|
| training, pre-training | adjusting the parameters, little by little, so the model gets things wrong less often. Pre-training is the long, general phase — reading a lot of text and learning the language — done before any specialisation | training people to use a system (IT management) | 00–06, 08–16 |
| fine-tuning | taking an already pre-trained model and continuing to train it, now with data from one specific task. Cheaper than starting from scratch, because it already knows the language | performance tuning of a database or an operating system (systems administration) | 00, 06, 08–12, 14, 15 |
| loss | the error score: a single number saying how wrong the model was, which training pushes down. When it stops falling, training stopped learning. Here it is computed with cross-entropy | packet loss (networking); lossy compression (multimedia) | 00, 08, 09 |
| entropy | how unsure the model is. Low when it bets almost everything on one word; high when it spreads the bet across many. In good morning, how are… entropy is low — you is nearly certain. Formally: the measure of uncertainty of a probability distribution, in Shannon's sense, counted in bits | the same mathematical sense, applied to the randomness an operating system gathers to generate random numbers and to the strength of a password (security) | 03, 04, 07, 08, 15 |
| gradient | training's compass: for each parameter, it says which way to move it, and how much, to reduce the loss. Technically, the vector of derivatives of the loss with respect to each parameter | a smooth transition between colours (design, CSS) | 02, 03, 05, 09, 15 |
| epoch | one complete pass through all the training material. Two epochs = the model saw every example twice | the Unix epoch, 1 January 1970, the zero point of time counting (operating systems) | 08, 10, 13 |
| batch | the handful of examples processed in one go, rather than one at a time. Doing it in groups uses the machine far better | batch processing: jobs run in sequence, without interaction (operating systems) | 00, 01, 05, 10, 14 |
| checkpoint | a copy of the weights saved partway through, so you can resume from there if something goes wrong, or compare versions and keep the best one | the point where the transaction log is written to disk (databases); a snapshot of a virtual machine's state (virtualization) | 13 |
| precision | here, how many bits hold each number: float32 uses 32, float16 uses 16. Fewer bits = less memory and more speed, with slightly less exactness. A very different sense from “precision” in statistics, which is beside it | a classification metric: of the cases marked positive, how many really were (statistics, system evaluation) | 02, 04, 11, 15 |
| logit | the raw score the model gives each possible word, before it becomes a percentage. It can be negative, and the scores do not add up to 100 — what turns them into odds is the softmax, the function that squeezes every score onto a 0-to-1 scale summing to 1 | in statistics, the function log(p / (1 − p)); neural-network usage extends the name to any score before normalization | 00, 02, 05–08, 12, 13, 15, 16 |
| term | in this course | in other areas of IT | chapters |
|---|---|---|---|
| inference | using the finished model to produce an answer. The opposite of training: no parameter changes, you only read what was already learned | deriving new facts from rules (expert systems, logic); type inference by the compiler (programming languages); a conclusion about a population from a sample (statistics) | 00, 05, 10, 14 |
| prompt | the text you hand the model to continue. It is not a command: it is the beginning of something it will complete | the command-line sign that waits for a command, such as C:\> or $ (operating systems) | 01, 05–09, 11, 15 |
| context | how much text the model can have in front of it at once. GPT-2 sees 1,024 tokens, and not one more — a limit built into its shape, not a setting | a context switch between processes on the CPU (operating systems); a program's execution context (programming) | 00–02, 04–12, 16 |
| temperature | the daring dial for writing. Near zero the model always picks the likeliest word and turns repetitive; above 1 it takes more risks and sometimes goes off the rails. Technically, the number that divides the logits before the softmax; the name comes from statistical physics | the operating temperature of a component, such as the CPU (hardware) | 00, 03, 07, 12, 14, 16 |
| sampling | drawing the next word according to the odds, instead of always taking the likeliest one. It is what makes the same prompt give different answers each time | the sampling rate of an audio signal (multimedia); a sample of a population in a survey (statistics) | 00, 05–08, 13, 15, 16 |
| cache | here, the KV cache: the model keeps the attention work already done for earlier words, so it does not redo everything for each new word. Without it, writing a long text gets unbearably slow | fast CPU memory; a local copy of HTTP responses; stored query results (hardware, web, databases). The principle is the same: keep it so it is not recomputed | 04, 06, 10, 11, 13–15 |
| kernel | a computation routine written to run fast on the graphics card — CUDA kernels. (In chapter 05 the word appears in another sense, the similarity function of an SVM: a classification method that predates deep neural networks) | the core of the operating system (operating systems) | 00, 05, 13, 14 |
| server, serving | “serving a model” means keeping it up, waiting for requests and answering — like any other service on the internet. It is the difference between having the model on disk and having the model in use | the same sense as a web or application server (networking); only what is served changes | 00, 13–15 |
The machine-learning definitions follow the field's general reference; the other meanings follow each discipline's standard or reference text.
Next: chapter 01 — Tokenization and BPE.
A term missing, or a meaning from another area that deserves to be here? The glossary grows with the course.