How vLLM Works?: PagedAttention, Continuous Batching and the KV Cache

Why vLLM matters?: PagedAttention, continuous batching, the KV cache and the performance trade-offs between latency, throughput, memory and cost. This article expands that outline

01The Problem: Memory, Not Arithmetic

Generating text with a transformer happens one token at a time. To produce each new token the model must attend to every earlier token in the conversation, and recomputing the attention keys and values for all of them at every step would be wasteful. So servers keep those keys and values in GPU memory and reuse them. That store is the KV cache.

The KV cache is what makes serving hard. The vLLM paper puts it this way: the cache for each request is huge and grows and shrinks dynamically, and when it is managed inefficiently, memory is wasted through fragmentation and duplication, which limits how many requests can be batched together. Fewer requests in a batch means lower throughput, because a GPU is far more efficient when it works on many sequences at once.

02The KV Cache: What It Costs

The cache stores a key and a value vector for every token, in every layer. Per token, that is roughly 2 (key and value) × layers × KV heads × head dimension × bytes per value. As an illustration, take a model with 32 layers, 8 KV heads and a head dimension of 128, stored in 16-bit precision:

Now add the naive approach to allocation. If a server reserves one contiguous buffer sized for the maximum sequence length, a request that has so far produced only 500 tokens in an 8,192-token slot has reserved memory that is about 94% unused, and nobody else can use it. Multiply that by dozens of requests and most of the GPU's spare memory is stranded.

03PagedAttention: Virtual Memory for the KV Cache

PagedAttention borrows an idea from operating systems. Instead of giving each request one contiguous region, it divides the KV cache into fixed-size blocks and gives each request a table that maps its logical sequence onto blocks scattered anywhere in memory. A request only takes a new block when its current one fills up.

The consequences are what the vLLM paper highlights: near-zero waste in KV-cache memory, because only the last block of a sequence can be partly empty, and flexible sharing of cache blocks within and across requests. Sharing matters when several requests start with the same text, as they do with a common system prompt, or when one prompt branches into several sampled outputs. The authors report 2–4× higher throughput than systems such as FasterTransformer and Orca at the same latency, with larger gains for longer sequences, bigger models and more complex decoding. That is the paper's own benchmark from 2023, so measure your own workload before relying on a multiple.

04Continuous Batching and the Scheduler

Classic static batching groups requests, runs them together, and waits for the slowest to finish before starting new work. Short requests sit idle behind long ones. Continuous batching schedules at the granularity of a single generation step: when a request finishes, its slot is refilled by a waiting request at the next step, so the GPU stays busy and the queue drains faster.

Two details from the vLLM documentation are worth knowing. First, chunked prefill: processing a long prompt (the compute-heavy prefill phase) can stall everyone else's token generation (the memory-bound decode phase). Chunked prefill splits large prefills into pieces and batches them alongside decode requests. In V1, the docs say it is enabled by default whenever possible, and the scheduler prioritises decode requests, then fills the remaining token budget with prefill work. Second, preemption: if the KV cache runs out of space, vLLM can preempt requests to free it, and the docs say V1's default preemption mode is recomputation, so a preempted request is recomputed later, which can hurt end-to-end latency.

05Prefix Caching: Do Not Recompute What You Already Know

Because the cache is already organised in blocks, vLLM can also reuse blocks across requests. Its automatic prefix caching hashes each KV-cache block using the tokens in the block and the tokens before it, so a new request that begins with the same prefix as an earlier one can reuse those blocks instead of recomputing them. The docs note that only full blocks are cached, and that the hash can include extra values such as LoRA identifiers, multimodal input hashes and cache salts that isolate caches in multi-tenant setups. In the current source on GitHub, prefix caching is enabled by default, though defaults change between versions, so check the one you run.

This helps most where many requests share a long prefix: a standard system prompt, retrieved documents in a RAG pipeline, or the growing history of a multi-turn chat.

06The Performance Trade-offs

There is no single speed number for an inference server. Five quantities pull against each other, and the settings that help one often cost another:

The documentation's tuning advice follows the same logic. Smaller values of max_num_batched_tokens give better inter-token latency because fewer prefills slow down decoding, larger values give better time to first token, and the docs suggest values above 8,192 for best throughput, especially for smaller models on large GPUs. If you see frequent preemptions, the docs suggest giving the cache more room (raise gpu_memory_utilization or tensor_parallel_size) or reducing concurrency (lower max_num_seqs or max_num_batched_tokens).

07Where vLLM Fits in an AI Architecture

vLLM is an engine, not a complete platform. It can serve as the inference layer behind an API gateway, connecting language models to applications such as RAG pipelines, chatbots and AI agents. It exposes an OpenAI-compatible server, so existing client code usually needs only a changed base URL.

The division of labour is clean. The gateway handles concerns such as authentication, rate limiting and routing, including choosing between models or replicas. vLLM handles inference and request scheduling. When one node is no longer enough, projects such as llm-d add routing that is aware of KV-cache contents and separate prefill and decode pools, running vLLM underneath. The Production AI Inference Stack article on this site covers that next layer.

08Putting It Together

vLLM's value comes from treating GPU memory as the scarce resource it is. PagedAttention keeps the KV cache compact and shareable, continuous batching keeps the GPU busy, chunked prefill keeps long prompts from starving everyone else, and prefix caching avoids repeating work. Together they let one GPU serve many users, and the trade-off knobs let you choose between a fast first token, a smooth stream and maximum throughput.

Further reading and grounding for the details referenced above: