From the function call to the HTTP server — and what production servers do that yours does not, and why that only matters past a certain scale.
Up to now, every lab script loaded the model from scratch. For gpt2-xl that costs 7 seconds and 5.8 GB of RAM — every single time.
An inference server solves this by inverting the relationship: the model stays loaded in a process that does not die, and anything talks to it over HTTP. You gain:
curl, a spreadsheet — anything that speaks HTTP..venv\Scripts\python.exe server\app.py -> http://localhost:8080/ dashboard -> http://localhost:8080/docs interactive API (Swagger)
It is ~200 lines in server/app.py, and it deliberately exposes the same endpoints as TGI and TEI — so that what you learn here transfers directly to the production servers.
| Endpoint | Origin | What it does |
|---|---|---|
POST /generate | TGI | completes a text; accepts the 5 models of the GPT family |
POST /generate_stream | TGI | the same, token by token, via Server-Sent Events |
POST /embed | TEI | 768-dimensional vector per text (normalized mean pooling) |
POST /predict | TEI | classification — the RoBERTa detector returning Fake/Real |
GET /info | TGI | available models and limits |
GET /health | TGI | liveness check |
# generate text curl -X POST http://localhost:8080/generate \ -H "Content-Type: application/json" \ -d '{"inputs": "The capital of France is", "model": "gpt2-medium", "parameters": {"max_new_tokens": 20, "do_sample": false}}' # response: # {"generated_text": " Paris, and the capital of France is Paris.", # "details": {"model": "gpt2-medium", "prompt_tokens": 5, # "generated_tokens": 15, "seconds": 2.649, # "tokens_per_second": 5.7}} # classify with the detector curl -X POST http://localhost:8080/predict \ -H "Content-Type: application/json" \ -d '{"inputs": "I went to the market and the tomatoes were a disgrace."}' # response: # [[{"label": "Fake", "score": 0.309467}, # {"label": "Real", "score": 0.690533}]]
Generate text with /generate using gpt2-large, and pass the result to /predict. That is how the numbers in chapter 12 were produced: the detector returned P(Fake) = 0.9997.
A decoder writes, an encoder judges, both in the same process, talking over HTTP. It is the whole course running in two requests.
text-generation-inference is Hugging Face's production server for generative models. Written in Rust (the router) plus Python (the engine), it went back to Apache 2.0 in April 2024, after a period under a restrictive license.
It does nothing conceptually different from our server. It does the same things much better, through four techniques worth knowing by name:
| Technique | The problem it solves |
|---|---|
| Continuous batching | Our server handles one request at a time. TGI groups requests from several users into a single batch — and, crucially, swaps requests in and out of the batch as some finish and others arrive, without waiting for the whole batch to end. It is the biggest throughput gain of the four. |
| PagedAttention | The KV cache (chapter 04) grows with every token and has an unpredictable size. Allocating it in contiguous blocks wastes enormous memory. PagedAttention manages it in pages, like an operating system's virtual memory. It came from the vLLM project. |
| Fused kernels | FlashAttention and the like: CUDA implementations that compute attention without ever materializing the |
| Tensor parallelism | Splitting a model too large for one GPU across several, with the layers sliced. |
TGI is distributed as a Linux Docker image and its engine depends on CUDA kernels. The lab was built for an ordinary machine: Windows, no NVIDIA GPU (an entry-level AMD card has no official ROCm support) and neither Docker nor WSL.
Getting TGI to run here would require installing WSL2 and Docker Desktop — a few hours and ~15 GB of disk, of which you have little to spare. And, in the end, you would have the same GPT-2 answering, inside a container, with no GPU to use the kernels that justify TGI.
The four techniques above only pay off from dozens of simultaneous requests upward. For one person learning, with one client at a time, they make no difference at all. That is why we wrote the same endpoints in 200 lines of Python — and the trade-off here favors learning, because you can read this whole server.
If one day you need to serve models in production, with an NVIDIA GPU and Linux: use TGI or vLLM. Do not write your own.
text-embeddings-inference is TGI's sibling for encoders, also Apache 2.0 since April 2024. And it brings a detail relevant to us:
Contrary to what is usually assumed, TEI does not serve only embeddings. It has a /predict endpoint for sequence classification models — exactly the category of our roberta-*-openai-detector. In other words: RoBERTa, which does not run on Ollama for lack of classification-head support (as chapter 00 explains), would run on TEI without any problem.
This stands as a correction of a common simplification: "encoders have no inference server" is false. The ecosystem has tooling for both branches of the tree — it is just that the encoder branch's tool is less famous.
TEI has a pure-Rust backend (candle) that can compile for CPU, so it is technically less hostile to Windows than TGI. But Windows is not a supported platform, and the compilation is a gamble. Our /predict does the same job today.
| Situation | Use |
|---|---|
| Learning, one user, CPU | this server — readable, no heavy dependencies |
| Prototype, few users | this server, or transformers directly |
| Production, decoder, NVIDIA GPU, Linux | vLLM or TGI |
| Production, embeddings or classification | TEI |
| Running an LLM on the desktop, no programming | Ollama or llama.cpp (but they do not serve encoders) |
| Maximum speed on CPU | llama.cpp with GGUF-quantized weights |
gpt2-xl generates at 3.6 tokens/s on this machine. A 100-token response takes ~28 seconds. Without streaming, the user stares at a frozen screen for 28 seconds. With streaming, they see the first word in 300 ms and the text growing.
The total time is identical. The perception is completely different — and that is why every AI assistant streams. In our server, the TextIteratorStreamer runs the generation in a separate thread and keeps emitting SSE events.
/embed vector comes outThe encoder returns one vector per token. To get one sentence vector, the average is taken over the real tokens (ignoring padding) and normalized to norm 1:
v = (h * mascara).sum(1) / mascara.sum(1) # mean over real tokens only v = F.normalize(v, dim=-1) # norm 1
The normalization is what allows comparing vectors by dot product — which, for norm-1 vectors, is cosine similarity. That is how vector databases do search quickly.
A note: we use mean pooling, not the <s>. For models trained specifically for embeddings (Sentence-BERT and derivatives), mean pooling is usually better. For classification, the <s> is used — as /predict does.
# terminal 1 — leave it running .venv\Scripts\python.exe server\app.py # terminal 2 .venv\Scripts\python.exe lab\11_api.py
/generate → /predict, measuring how strongly the detector flags it./embed — and fails. Read the box below before running; that failure is the most instructive part of the lab.I assembled a collection of 6 sentences and asked "Where is the famous iron tower located?". The system returned the sentence about the Amazon rainforest. And most revealing: all the similarities landed between +0.99 and +1.00. According to these vectors, the Eiffel Tower and the Amazon are 99.8% alike.
This is called anisotropy. Instead of spreading over the 768-dimensional sphere, the vectors collapse into a narrow cone — all pointing in almost the same direction. The cosine loses its power to discriminate.
Centering helps partially. Subtracting the mean of the vectors before comparing, the scores spread out again (from +0.99 to a range of +0.54 to +0.79) and the system starts getting some of the questions right. But it still gets things wrong — because this treats the symptom.
The causes are two, stacked:
roberta-base-openai-detector, fine-tuned for a binary task. The tuning reorganized the representations around the "human vs machine" axis and discarded semantic structure that was of no use for that.It was precisely to solve this that Sentence-BERT was created (Reimers & Gurevych, 2019): take a BERT and train it with a similarity objective, using pairs of similar and dissimilar sentences. Embedding models such as E5, BGE and gte do some variant of this.
The lesson: "it is an encoder" does not mean "it works for embeddings". The training objective determines what the representations are good for. The mechanics the lab shows (vectorize → normalize → dot product → sort) are correct and are those of any vector database. Swap the model for one trained for embeddings and it works.
http://localhost:8080/docs and fire off a generation through the form. Change temperature and seed and observe what happens to reproducibility.seed and do_sample=true, does the output repeat? And with a null seed? Why?/generate loads the model on the first call and keeps it cached. Make two calls to gpt2-xl and compare the times. Where did the 7 seconds go?The primary sources for the concepts in this chapter. The numbers in the text were measured in the lab; the ideas come from here.
Technical terms here carry their machine-learning meaning. The course glossary shows that meaning next to the others the same words have in IT.
Chapter 15 is the reference map: every script, every command, every file — the guide for you to come back to later.