Serving models: from function call to HTTP server, and what production does on top
Our server, TGI, TEI, and the scale from which each difference matters.
Landed here from LinkedIn? This is chapter 14 of the course GPT-1 and GPT-2, taken apart. It reads on its own; if you want the beginning, it is chapter 00.
Loading GPT-2 XL costs 7 seconds and 5.8 GB of RAM. Every single time. Every lab script loaded the model from scratch. An inference server inverts the relationship: the model sits in a process that does not die and anything talks to it over HTTP. We wrote one in 200 readable lines, with the same endpoints as Hugging Face's production servers. What the production ones have and ours does not are exactly the four things that matter from thousands of requests per second upward, and that do not matter when you are learning.
The three points
- A FastAPI server with TGI's and TEI's endpoints: generate, generate_stream, embed, predict. Compatible, readable, no GPU.
- Continuous batching, paged attention, tensor parallelism, quantisation: what TGI has, what each solves, and when you need it.
- Good news about RoBERTa: for embeddings and classification, it runs in milliseconds on a CPU.
From the chapter, the beginning
From the function call to the HTTP server — and what production servers do that yours does not, and why that only matters past a certain scale.
Up to now, every lab script loaded the model from scratch. For gpt2-xl that costs 7 seconds and 5.8 GB of RAM — every single time.
What the chapter covers · 7 sections · 9 min
- Why serve over HTTP
- Our server
- What TGI really is
- TEI and some good news about RoBERTa
- The decision table
- Implementation details worth reading
- In the lab
All chapters of the course · GPT-1 and GPT-2, taken apart
- 00 · The seven fossils: why GPT-1, GPT-2 and RoBERTa are a historical cross-section, not a list
- Glossary · Course glossary: what token, layer and model mean in AI, and in the rest of IT
- 01 · Why GPT-2 sees neither letters nor words, and what that costs in Portuguese
- 02 · Embeddings: from a meaningless integer to a vector that knows king and queen are related
- 03 · Attention: the single operation that replaced twenty years of recurrent networks
- 04 · Twelve heads instead of one, and the single line of code that separates GPT from RoBERTa
- 05 · Why GPT-2 needs 48 floors to answer, and what happens inside each one
- 06 · What scale really buys: from 124 million to 1.5 billion, measured
- 07 · Temperature, top-k and top-p: the decision that changes the text more than switching models
- 08 · Perplexity: the one number that says what the model learned, without anecdotes
- 09 · GPT-1 to GPT-2: same size, half the perplexity. Four decisions explain it
- 10 · RoBERTa: when the advance is training better what already existed
- 11 · Encoder vs decoder: one line of code and every consequence that descends from it
- 12 · The detector: how a language model becomes a classifier, and why this one should decide nothing
- 13 · Reading GPT-2's original code: 174 lines against everything you learned
- 14 · Serving models: from function call to HTTP server, and what production does on top
- 15 · The lab guide: every command, every script, and everything that was measured
- 16 · The ceiling of this path: LeCun's argument, read from inside GPT-2
Writes the GPT-2 course and runs the Antivision experiment, measuring every run before publishing the number. The LinkedIn posts start here and end in the chapters.
Follow on LinkedIn →