LLM Training and Inferencing Optimization Techniques

The simplest and most foundational optimization: stop recomputing what you've already computed.

During each step of autoregressive text generation, an LLM compares the last token's Query vector against the Key vectors of every preceding token, including itself. This produces attention weights, which are used to compute a weighted sum over the corresponding Value vectors — the attention output for that token.

For example, when generating the token that follows "The capital of the U.K. is", the model:

• Converts the current token ("is") into a Query vector
• Scores it against the Key of every preceding token, finding that "capital" and "U.K." are most relevant
• Combines those tokens' Value vectors to predict that the next token is most likely "London"

This repeats at every step of generation. Recomputing Key and Value vectors for the entire sequence at each step gets expensive fast as the sequence grows — which is exactly the problem a KV cache solves.

Instead of recalculating Key and Value vectors for the whole sequence every time, the KV cache stores each token's K and V vectors once. At each new step, the model only computes (and appends) the new token's Key and Value, reusing everything else from the cache.

This massively reduces inference compute — but at the cost of GPU high-bandwidth memory (HBM). There's no free lunch. Still, KV caching is one of the simplest, highest-leverage techniques for improving inference speed.

Since the KV cache eats into HBM, shrinking it is its own optimization target.

Traditional Multi-Head Attention (MHA) gives every attention head its own Key and Value vectors. Several attention variants reduce the number of K/V vectors in use — and therefore the KV cache size:

• Multi-Query Attention (MQA) — all attention heads share one Key and Value, while each head keeps its own Query.
• Grouped Query Attention (GQA) — Query heads are grouped, and each group shares one Key/Value pair.
• Multi-head Latent Attention (MLA) — Key and Value vectors are compressed into a low-dimensional latent vector and only that is cached. The full K/V vectors are reconstructed from the latent vector when needed, using far less memory than a full-sized KV cache.

Processing requests one at a time wastes GPU compute — batching fixes that, but how you batch matters.

Handling LLM requests sequentially is inefficient, since requests arrive at different times and have variable input/output lengths. Batching requests together improves overall throughput and reduces idle GPU compute — but there's more than one way to do it:

• Static batching — the system waits for a fixed number of requests to arrive before processing them together.
• Dynamic batching — the system waits only up to a maximum time limit rather than indefinitely, before starting the batch.
• Continuous (in-flight) batching — processing happens token by token. Instead of waiting for the whole batch to finish, the system evicts a completed request and immediately fills its slot with a new one.

Decode is memory-bound — the bottleneck is moving data, not computing on it.

During inference, the Decode phase is memory-bound: the GPU spends more time moving data from memory (HBM) to its compute cores than actually computing. A kernel is a function that runs on the GPU performing parallel operations, and it can be optimized to keep data in fast on-chip memory (SRAM/registers) rather than repeatedly re-reading it from slow HBM.

Kernel fusion merges multiple individual operations into a single kernel to avoid unnecessary data movement across memory. FlashAttention is the best-known example, fusing the operations of attention computation into one kernel.

Standard attention computation moves data back and forth repeatedly: Query and Key are read from HBM and multiplied to form an attention score matrix, which is written back to HBM, re-read to apply softmax, written back again, then read once more to multiply by the Value matrix — with the final output written back to HBM one last time.

FlashAttention avoids all of this by never materializing the full attention matrix in HBM. It splits the Query, Key, and Value matrices into smaller tiles that fit in on-chip SRAM, and performs attention one tile at a time — computing attention scores and softmax on-chip, and writing only the final output back to HBM.

Store and compute in fewer bits, and both memory footprint and throughput improve.

Quantization stores and computes LLM parameters, activations, and the KV cache in lower-precision formats such as INT8 (8 bits) or INT4 (4 bits) instead of full-precision FP32 or 16-bit FP16/BF16. This shrinks memory requirements and increases throughput by reducing how much data moves from HBM to the compute cores — at the cost of a slight accuracy drop.

Quantized models are commonly labeled using the convention W(x)A(y), describing how much the weights (W) and activations (A) have been quantized:

You'll also see model names suffixed with -GPTQ or -AWQ — the two most common quantization techniques:

• GPTQ — Generative Pre-trained Transformer Quantization
• AWQ — Activation-Aware Weight Quantization

Many requests share the same opening tokens — compute that shared prefix once.

LLM inference has two distinct phases:

• Prefill — all input prompt tokens are processed together, which is compute-bound. Prefill determines Time-to-First-Token (TTFT), and it's when the KV cache is built.
• Decode — subsequent tokens are generated one at a time, which is memory-bound since little computation happens per step. Decode determines Time Per Output Token (TPOT).

Prefix (or prompt) caching reuses the computed KV cache for a shared prefix of the input prompt across multiple requests, cutting redundant Prefill computation and shortening TTFT. Consider a long system prompt like "System: You are a helpful assistant..." — without prefix caching, every request recomputes Prefill for the entire system prompt plus the user's question. With it, the system prompt's KV cache is computed once and reused; each request only computes Prefill for its own unique question.

RadixAttention, introduced with the SGLang serving framework, implements prefix caching efficiently using a radix tree: each path from the root represents a token sequence, and shared prefixes across requests become shared branches. For a new request, the algorithm walks the tree, matching tokens as far as possible and reusing the KV cache along that path — computing Prefill only for the tokens unique to that request.

Not every parameter earns its keep — pruning removes the ones that don't.

Pruning removes parts of a model that contribute little to its accuracy, making it smaller and cheaper to run inference on. It comes in three flavors:

1. Unstructured pruning — sets low-magnitude individual parameters to zero.
2. Structured pruning — removes entire sections of the model, such as neurons, attention heads, or fully-connected layers.
3. Semi-structured (N:M) pruning — sets N of every M consecutive weights to zero. NVIDIA GPUs from Ampere onward have hardware support for 2:4 sparsity, giving roughly a 2x boost in matrix multiplication throughput.

Borrowed from operating systems: don't demand one giant contiguous block of memory.

PagedAttention, introduced in the vLLM serving framework, manages the KV cache's memory more efficiently by borrowing an idea from how operating systems handle memory paging.

Instead of reserving one large contiguous chunk of GPU memory for the KV cache upfront, PagedAttention stores it dynamically across many small, fixed-size, non-contiguous blocks called pages. A per-sequence block table maps logical token positions to their physical memory blocks, and the attention kernel consults this table to locate the right KV pairs on demand.

Use a fast, small model to guess ahead — then let the big model verify the guesses in one pass.

Speculative decoding accelerates inference with large LLMs by pairing them with a smaller model from the same family, sharing the same tokenizer and vocabulary. The smaller model — the Draft model — runs faster than the larger Target model simply because it has fewer parameters. The process works like this:

1. Drafting — for a given prompt, the Draft model generates K tokens ahead of time.
2. Target forward pass — those K draft tokens, plus the original prompt, are fed to the Target model. In a single forward pass, it produces a probability distribution for the next token at every drafted position, plus one more beyond it (K + 1 predictions total). Because Decode is memory-bound, this single pass costs about as much as generating just one token normally would.
3. Verification — each drafted token is checked against the Target model's prediction at that position. It's accepted if they match (under greedy decoding) or if the Target model finds it likely enough via rejection sampling (under probabilistic decoding). At the first token where verification fails, that token and every token after it are discarded, the Target model supplies a replacement, and drafting resumes from that position.

Prefill and Decode want different things from hardware — so give them different hardware.

Prefill is compute-bound and determines TTFT; Decode is memory-bound and determines TPOT (also called inter-token latency). These are fundamentally different problems, and optimizing both on the same GPU is difficult — especially for prefill-heavy requests where long prompts produce short outputs, and low TTFT and low TPOT are both required at once.

Prefill-decode disaggregation is a serving architecture that solves this by moving the two inference stages onto separate, dedicated GPU pools: one pool optimized for high compute (e.g. the H100) handles Prefill, while a pool optimized for high memory throughput (e.g. the H200, which offers much faster and larger memory at similar core compute) handles Decode.

Prefill vs. decode. Prefill processes the prompt and builds the KV cache; it parallelizes well and is usually compute-bound. Decode generates one token at a time and is often limited by memory bandwidth at practical batch sizes. Most techniques below target one phase or the other.

• FlashAttention reduces memory traffic inside attention; PagedAttention manages the KV cache in blocks to cut fragmentation.
• MQA and GQA use fewer key/value heads, shrinking KV-cache storage and bandwidth.
• Continuous batching lets requests join and leave the active batch mid-generation; chunked prefill splits long prompts so they can run alongside decode work.
• Prefix caching reuses KV state for repeated prompt prefixes. It saves prefill compute but does not speed up later decode tokens, and it matters most for agentic workloads that resend growing prefixes.
• Quantization applies to weights (less memory and bandwidth) and to the KV cache (smaller footprint).
• Speculative decoding pays off only when the draft is cheap, acceptance rate is high and verification is efficient.
• Parallelism at scale: tensor parallelism adds communication overhead, pipeline parallelism adds scheduling bubbles, and mixture-of-experts serving adds routing and load-balancing problems.
• Prefill/decode disaggregation runs the phases on separate GPU pools at the cost of moving KV state between them.

Beyond performance, production serving also needs access control, cost tracking, token budgets and guardrails; avoiding a request altogether through caching and routing can be cheaper than optimizing GPU throughput.

The single highest-leverage decision in the whole pipeline happens before a single token is generated: which model handles this request at all. Frontier models are priced for the hardest 10-15% of tasks a system will ever see, but most production traffic is routine — classification, extraction, short summarization, templated drafting — and never needed frontier-level reasoning in the first place. A router that estimates request complexity and dispatches accordingly turns that observation into savings without touching output quality on the requests that actually matter.

This isn't theoretical. RouteLLM, a widely cited open routing framework, reported 85% cost savings while retaining 95% of GPT-4-level quality on MT-Bench, sending only 14% of queries to the stronger model. Field reports from teams running tuned routers in production land in a broad 40-85% bill-reduction range, and the math is intuitive once you model it as a traffic split: routing 70% of volume to a cheap model and 30% to a frontier model (say, Haiku-class to Opus-class) nets roughly 67% savings versus sending everything to the frontier tier; an 80/20 split nets closer to 79%. The savings compound sharply once the cheap-model share crosses 50% of total volume — which is exactly where most real workloads sit once you actually measure them.

Hybrid cloud/on-prem routing is the same idea applied to a different axis: instead of routing by model size, route by data sensitivity and predictability. Workloads with stable, well-understood shapes — and anything where sending data off-premises is itself the cost (compliance risk, egress fees, latency to a regulated data residency zone) — stay on local or on-prem inference. Genuinely open-ended reasoning still goes to a cloud frontier model when the task warrants it. Both forms of routing rest on the same principle: complexity and sensitivity, not habit, should decide where a request lands.

Routing picks the model; the serving engine underneath still decides how much of a GPU's capacity a request wastes. Continuous batching and PagedAttention, covered above as techniques 3 and 8, are the two changes that matter most here.

Caching shows up at three different layers of the stack, and conflating them is the most common mistake in a cost-optimization plan. Each one catches a different kind of repetition.

Prefix (prompt) caching

Prefix caching operates inside the model's attention mechanism: it stores KV-cache entries for a prompt's beginning and reuses them when a new request's prefix matches exactly, token for token. It only reduces time-to-first-token — output generation still runs, and still costs, at full price. But the TTFT effect scales with prefix length: roughly 7% improvement at a 1,024-token shared prefix, climbing to 67% at 150,000+ tokens, and around 79% TTFT reduction with a 90% cost cut on the cached portion at a 100,000-token cached prefix in benchmark testing. This is the mechanism behind the "prompt caching" discounts offered by Anthropic, OpenAI, Google, and AWS Bedrock.

Semantic caching

Semantic caching sits a layer above the model entirely. Incoming queries are converted to embeddings and compared against cached queries by cosine similarity against a configurable threshold; on a hit, the LLM is never called at all — both input and output tokens are fully saved, not just discounted. Because it matches on meaning rather than exact text, it catches paraphrased and reworded repeats that prefix caching can't touch, which is why teams report meaningfully higher hit rates and total savings in the range of 30-40%+ of LLM spend when it's tuned well. The trade-off is a small risk of a false-positive match returning a stale or subtly wrong cached answer, which is why threshold tuning and cache invalidation policy matter more here than in exact-match caching.

The two are complementary, not competing — a layered strategy runs exact-match caching first, semantic caching second, and prefix caching for whatever's left that still has to reach the model.

KV cache compression

Independent of caching strategy, the KV cache itself can be shrunk architecturally. Grouped-Query Attention (GQA) delivers 4-8x compression and is the default in Llama 3.x and Mistral; Multi-Query Attention (MQA) pushes to 32x compression at a noticeable 1-3 point quality cost; Multi-head Latent Attention (MLA), used in DeepSeek's V2 through V4 models, achieves 7-14x compression with under 0.2 points of regression. Layered on top, FP8 KV-cache quantization halves memory again for a modest 0.3-0.7 point hit. Stacked together, these compound multiplicatively — a 70B model at 1M-token context can go from 135GB of KV cache at FP16 baseline down to roughly 8GB with MLA plus FP8, a documented 4-40x reduction on long-context inference depending on which techniques are combined.

Routing and caching reduce how often and how much of the model runs. Quantization and distillation reduce the cost of running the model itself.

Quantization lowers the numeric precision of model weights (and optionally activations and the KV cache) from 16-bit down to 8-bit or 4-bit representations. The two dominant post-training methods differ in how they decide what to protect: GPTQ relies on layer-wise second-order weight optimization, which is computationally expensive and less targeted; AWQ (Activation-Aware Weight Quantization) instead profiles activation magnitudes during calibration to identify the roughly 1% of "salient" weight channels that matter most, scales those channels to preserve precision, and quantizes the rest aggressively — and it consistently outperforms GPTQ by 1-2 quality points at the same bit-width. The economics are substantial: a 70B model drops from roughly 140GB in BF16 to 35-40GB in AWQ INT4, a ~71% memory reduction, which in one benchmark comparison took serving cost from $0.757 per million tokens on dual A100s down to $0.253 per million tokens on a single A100 — a ~67% cost reduction — with throughput improving roughly 1.5x on top of that.

Distillation and task-specific fine-tuning attack the same problem from a different angle: instead of shrinking a general-purpose model's weights, train a much smaller student model to reproduce a larger teacher's behavior on a narrower task. This is the right call precisely when prompts are long, repetitive, and narrow in scope — sending the same lengthy context and instructions on every call is often more expensive over time than the one-time cost of fine-tuning a small model to do that one job without the context at all. The trade-off is scope: a distilled or fine-tuned model is deliberately worse at everything outside its training distribution, which is why it belongs behind a router, not in front of general traffic.

Everything above reduces how much compute a request costs. This section is about latency — and, indirectly, cost, since lower latency means fewer retries and less GPU time held open per request.

Speculative decoding is covered above (technique 9): a small draft model proposes tokens that the full model verifies in one pass.

Asynchronous inference is the simpler lever: not every workload needs a synchronous, sub-second response. Batch scoring, overnight report generation, and background enrichment jobs can be queued and processed during off-peak capacity, which both improves GPU utilization and often qualifies for lower off-peak or batch API pricing where providers offer it.

Streaming responses address perceived latency rather than actual compute cost: delivering tokens as they're generated, rather than waiting for the full response, keeps a user or downstream system engaged instead of timing out or retrying — and retries are pure wasted spend, since a timed-out request that gets resent pays for the discarded generation too.

Every token in the context window is compute you're paying for whether or not it helps the answer — and, on current transformer architectures, attention cost scales with context length, so a bloated context window is a compounding cost, not a flat one. Context window auditing means treating retrieval and prompt assembly as a relevance-filtering problem: pull in what a request needs, not everything a retrieval system could plausibly return. This isn't only a cost argument — models are also demonstrably worse at using information buried in the middle of an very long context than information near the start or end, so an aggressively padded prompt can simultaneously cost more and perform worse.

Output token controls close the other end of the same problem. An unbounded max_tokens setting, combined with a model that tends to over-explain, restate, or hedge, can quietly become one of the largest line items in an inference bill — and it's one of the cheapest to fix, since explicit output limits and stop sequences cost nothing to implement and directly cap the most expensive part of a request (output tokens are typically priced several times higher than input tokens across most providers).

Every technique above is a lever inside the request path. This last pair operates outside it, and it's the layer that keeps the others from silently drifting back toward waste.

Usage governance — quotas, per-team and per-application budgets, rate limits, and hard guardrails on which model tiers a given caller can even reach — has to exist before a runaway integration or an unbounded agent loop turns into a five-figure surprise, not after. A router that's technically capable of sending everything to a frontier model will do exactly that if nothing stops it, whether the cause is a misconfigured default or a legitimate but unbounded batch job.

Cost and usage observability is what makes every other technique in this piece measurable rather than assumed. Tracking spend by application, feature, team, workflow, model, and user segment turns "our AI costs went up" into "feature X's fallback path is calling the frontier model on cache misses at 3x the volume we expected" — an actionable finding instead of a vague budget alarm. This is also the feedback loop that closes back into routing and caching: a router's complexity thresholds and a semantic cache's similarity threshold are not "set once" parameters — they need retuning as traffic mix shifts, and that retuning is only possible with granular, current cost data to look at.