LLM Inference Optimization Techniques

The simplest and most foundational optimization: stop recomputing what you've already computed.

During each step of autoregressive text generation, an LLM compares the last token's Query vector against the Key vectors of every preceding token, including itself. This produces attention weights, which are used to compute a weighted sum over the corresponding Value vectors — the attention output for that token.

For example, when generating the token that follows "The capital of the U.K. is", the model:

• Converts the current token ("is") into a Query vector
• Scores it against the Key of every preceding token, finding that "capital" and "U.K." are most relevant
• Combines those tokens' Value vectors to predict that the next token is most likely "London"

This repeats at every step of generation. Recomputing Key and Value vectors for the entire sequence at each step gets expensive fast as the sequence grows — which is exactly the problem a KV cache solves.

Instead of recalculating Key and Value vectors for the whole sequence every time, the KV cache stores each token's K and V vectors once. At each new step, the model only computes (and appends) the new token's Key and Value, reusing everything else from the cache.

This massively reduces inference compute — but at the cost of GPU high-bandwidth memory (HBM). There's no free lunch. Still, KV caching is one of the simplest, highest-leverage techniques for improving inference speed.

Since the KV cache eats into HBM, shrinking it is its own optimization target.

Traditional Multi-Head Attention (MHA) gives every attention head its own Key and Value vectors. Several attention variants reduce the number of K/V vectors in use — and therefore the KV cache size:

• Multi-Query Attention (MQA) — all attention heads share one Key and Value, while each head keeps its own Query.
• Grouped Query Attention (GQA) — Query heads are grouped, and each group shares one Key/Value pair.
• Multi-head Latent Attention (MLA) — Key and Value vectors are compressed into a low-dimensional latent vector and only that is cached. The full K/V vectors are reconstructed from the latent vector when needed, using far less memory than a full-sized KV cache.

Processing requests one at a time wastes GPU compute — batching fixes that, but how you batch matters.

Handling LLM requests sequentially is inefficient, since requests arrive at different times and have variable input/output lengths. Batching requests together improves overall throughput and reduces idle GPU compute — but there's more than one way to do it:

• Static batching — the system waits for a fixed number of requests to arrive before processing them together.
• Dynamic batching — the system waits only up to a maximum time limit rather than indefinitely, before starting the batch.
• Continuous (in-flight) batching — processing happens token by token. Instead of waiting for the whole batch to finish, the system evicts a completed request and immediately fills its slot with a new one.

Decode is memory-bound — the bottleneck is moving data, not computing on it.

During inference, the Decode phase is memory-bound: the GPU spends more time moving data from memory (HBM) to its compute cores than actually computing. A kernel is a function that runs on the GPU performing parallel operations, and it can be optimized to keep data in fast on-chip memory (SRAM/registers) rather than repeatedly re-reading it from slow HBM.

Kernel fusion merges multiple individual operations into a single kernel to avoid unnecessary data movement across memory. FlashAttention is the best-known example, fusing the operations of attention computation into one kernel.

Standard attention computation moves data back and forth repeatedly: Query and Key are read from HBM and multiplied to form an attention score matrix, which is written back to HBM, re-read to apply softmax, written back again, then read once more to multiply by the Value matrix — with the final output written back to HBM one last time.

FlashAttention avoids all of this by never materializing the full attention matrix in HBM. It splits the Query, Key, and Value matrices into smaller tiles that fit in on-chip SRAM, and performs attention one tile at a time — computing attention scores and softmax on-chip, and writing only the final output back to HBM.

Store and compute in fewer bits, and both memory footprint and throughput improve.

Quantization stores and computes LLM parameters, activations, and the KV cache in lower-precision formats such as INT8 (8 bits) or INT4 (4 bits) instead of full-precision FP32 or 16-bit FP16/BF16. This shrinks memory requirements and increases throughput by reducing how much data moves from HBM to the compute cores — at the cost of a slight accuracy drop.

Quantized models are commonly labeled using the convention W(x)A(y), describing how much the weights (W) and activations (A) have been quantized:

You'll also see model names suffixed with -GPTQ or -AWQ — the two most common quantization techniques:

• GPTQ — Generative Pre-trained Transformer Quantization
• AWQ — Activation-Aware Weight Quantization

Many requests share the same opening tokens — compute that shared prefix once.

LLM inference has two distinct phases:

• Prefill — all input prompt tokens are processed together, which is compute-bound. Prefill determines Time-to-First-Token (TTFT), and it's when the KV cache is built.
• Decode — subsequent tokens are generated one at a time, which is memory-bound since little computation happens per step. Decode determines Time Per Output Token (TPOT).

Prefix (or prompt) caching reuses the computed KV cache for a shared prefix of the input prompt across multiple requests, cutting redundant Prefill computation and shortening TTFT. Consider a long system prompt like "System: You are a helpful assistant..." — without prefix caching, every request recomputes Prefill for the entire system prompt plus the user's question. With it, the system prompt's KV cache is computed once and reused; each request only computes Prefill for its own unique question.

RadixAttention, introduced with the SGLang serving framework, implements prefix caching efficiently using a radix tree: each path from the root represents a token sequence, and shared prefixes across requests become shared branches. For a new request, the algorithm walks the tree, matching tokens as far as possible and reusing the KV cache along that path — computing Prefill only for the tokens unique to that request.

Not every parameter earns its keep — pruning removes the ones that don't.

Pruning removes parts of a model that contribute little to its accuracy, making it smaller and cheaper to run inference on. It comes in three flavors:

1. Unstructured pruning — sets low-magnitude individual parameters to zero.
2. Structured pruning — removes entire sections of the model, such as neurons, attention heads, or fully-connected layers.
3. Semi-structured (N:M) pruning — sets N of every M consecutive weights to zero. NVIDIA GPUs from Ampere onward have hardware support for 2:4 sparsity, giving roughly a 2x boost in matrix multiplication throughput.

Borrowed from operating systems: don't demand one giant contiguous block of memory.

PagedAttention, introduced in the vLLM serving framework, manages the KV cache's memory more efficiently by borrowing an idea from how operating systems handle memory paging.

Instead of reserving one large contiguous chunk of GPU memory for the KV cache upfront, PagedAttention stores it dynamically across many small, fixed-size, non-contiguous blocks called pages. A per-sequence block table maps logical token positions to their physical memory blocks, and the attention kernel consults this table to locate the right KV pairs on demand.

Use a fast, small model to guess ahead — then let the big model verify the guesses in one pass.

Speculative decoding accelerates inference with large LLMs by pairing them with a smaller model from the same family, sharing the same tokenizer and vocabulary. The smaller model — the Draft model — runs faster than the larger Target model simply because it has fewer parameters. The process works like this:

1. Drafting — for a given prompt, the Draft model generates K tokens ahead of time.
2. Target forward pass — those K draft tokens, plus the original prompt, are fed to the Target model. In a single forward pass, it produces a probability distribution for the next token at every drafted position, plus one more beyond it (K + 1 predictions total). Because Decode is memory-bound, this single pass costs about as much as generating just one token normally would.
3. Verification — each drafted token is checked against the Target model's prediction at that position. It's accepted if they match (under greedy decoding) or if the Target model finds it likely enough via rejection sampling (under probabilistic decoding). At the first token where verification fails, that token and every token after it are discarded, the Target model supplies a replacement, and drafting resumes from that position.

Prefill and Decode want different things from hardware — so give them different hardware.

Prefill is compute-bound and determines TTFT; Decode is memory-bound and determines TPOT (also called inter-token latency). These are fundamentally different problems, and optimizing both on the same GPU is difficult — especially for prefill-heavy requests where long prompts produce short outputs, and low TTFT and low TPOT are both required at once.

Prefill-decode disaggregation is a serving architecture that solves this by moving the two inference stages onto separate, dedicated GPU pools: one pool optimized for high compute (e.g. the H100) handles Prefill, while a pool optimized for high memory throughput (e.g. the H200, which offers much faster and larger memory at similar core compute) handles Decode.