How LLM Inference Works
AdvancedLLM inference is what happens when a trained model generates a response. The input text is split into tokens, the model processes them, and it produces output one token at a time, each new token predicted from all the tokens so far. This autoregressive generation has two phases — a parallel prefill of the prompt and a sequential decode of the output — and relies on a KV cache to avoid recomputing past work. Understanding inference explains why LLMs cost what they do, why the first token is slower, and how batching and quantization cut cost.
Think of writing a sentence one word at a time, never erasing
An LLM writes like someone composing a sentence word by word, where each next word depends on everything written so far, and nothing can be un-written. First they read the whole prompt (fast, all at once — prefill). Then they add words one at a time, re-reading the growing text before each new word (decode). To avoid re-reading from scratch every time, they keep notes on what they have already processed (the KV cache), so each new word only requires a little extra thought.
Step by Step
Key Concepts
Tokens
The units a model reads and writes — word pieces, not characters or whole words. Context limits, latency, and cost are all measured in tokens, so token count directly drives price and speed.
Prefill vs Decode
Prefill processes the whole prompt in parallel and sets the time-to-first-token; decode generates output tokens one at a time and sets the per-token speed. They have very different performance characteristics.
KV Cache
Stored key/value tensors for already-processed tokens so each new token does not reprocess the whole sequence. It makes decoding efficient but grows with sequence length and dominates GPU memory during serving.
Batching
Serving multiple requests together to raise GPU utilisation. Because decode is memory-bound, batching greatly increases throughput (and lowers cost per request), which is why hosted APIs batch aggressively.
Key Facts
- Generation is autoregressive and sequential — output tokens are produced one at a time — which is why longer responses take proportionally longer and cost more.
- The first token is slower than subsequent ones because it waits for prefill; time-to-first-token and tokens-per-second are the two latency numbers that matter.
- Decode is memory-bandwidth-bound, not compute-bound, so batching many requests and quantizing weights are the main ways to improve throughput and cut serving cost.
Real-World Applications
Serving a chat product cost-effectively
A team improves throughput by batching concurrent requests and quantizing the model, and manages the KV cache carefully, dramatically lowering the GPU cost per conversation without noticeably hurting quality.
Streaming responses to users
Because tokens are generated one at a time, apps stream them to the UI as they are produced, so users see the answer appear progressively rather than waiting for the whole response.
Frequently Asked Questions
What is LLM inference?
LLM inference is the process of using a trained large language model to generate a response to an input. The input text is tokenized, the model processes it, and it produces output one token at a time, with each new token predicted from all the tokens so far. Inference is distinct from training: it runs the finished model to serve requests, and its efficiency determines the latency and cost of every response.
What are the prefill and decode phases?
Prefill is the first phase, where the model processes the entire input prompt in parallel to build up its internal state; it is compute-heavy and determines the time to the first output token. Decode is the second phase, where the model generates the output autoregressively — one token at a time, each depending on all previous tokens. Decode is sequential and sets the per-token generation speed. The two phases have very different performance profiles, which is why the first token often takes noticeably longer than subsequent ones.
What is the KV cache in LLM inference?
The KV cache stores the key and value tensors the model computes for tokens it has already processed. Without it, generating each new token would require reprocessing the entire sequence from scratch, which is prohibitively expensive. With the cache, each new token reuses the stored state and only computes what is needed for itself, making decoding efficient. The trade-off is that the KV cache grows with sequence length and consumes a large share of GPU memory during serving.
How do you make LLM inference cheaper and faster?
The main levers are batching and quantization. Because the decode phase is limited by memory bandwidth rather than compute, serving many requests together (batching) raises GPU utilisation and throughput, lowering cost per request. Quantization uses lower-precision weights to shrink memory usage and speed up inference with minimal quality loss. Managing the KV cache efficiently and choosing appropriately sized models for the task also significantly affect cost and latency.