How Modern AI Clusters Load Balance GPU Inference for LLMs

Every LLM request passes through a prefill phase (processing the prompt into key-value tensors) and a decode phase (generating tokens one at a time from that cache). If 100 users share the same system prompt or few-shot context, routing all of them to the replica that already holds that prefix in VRAM skips redundant prefill work entirely — vendor benchmarks report this can eliminate the large majority of repeated prefill computation for high-overlap workloads.

In practice, prefix-aware routing is implemented a few different ways: session affinity by client identity, consistent hashing of the prompt prefix so identical prefixes land on the same replica, an approximate cache map maintained by the router, or — as in llm-d — precise cache metadata streamed from the inference engines themselves (vLLM and SGLang both emit KV-cache events the router can subscribe to). Good routers also track cache occupancy, not just hit probability: a replica whose cache is nearly full may evict useful blocks under pressure, so it can be worth accepting a smaller cache hit on a replica with more headroom.

Prefill and decode have opposite hardware profiles: prefill is compute-bound (it wants raw FLOPS to digest the prompt fast), while decode is memory-bandwidth-bound (it wants to stream one token at a time as fast as VRAM can be read). Running both on the same GPU pool forces a compromise. Modern inference engines instead split the cluster into dedicated prefill workers and decode workers, each provisioned and scaled for its own bottleneck, with a router/transfer layer moving the computed KV cache from the prefill worker to the decode worker that continues the generation.

Because that KV-cache handoff is itself bandwidth-sensitive, disaggregated deployments generally place paired prefill/decode workers close together on the network — often within the same rack over NVLink or another high-bandwidth interconnect — to keep the transfer from becoming its own bottleneck.

Speculative decoding pairs a small, fast draft model that proposes several candidate tokens at once with a larger target model that verifies them in a single parallel pass — accepting the draft tokens that match and falling back to normal generation the moment one doesn't. Because the draft and target models have very different resource profiles, some deployments route them to different hardware tiers entirely: lightweight draft inference on smaller or edge-adjacent GPUs, with verification against the full model concentrated on centralized, high-memory GPU clusters.

This is a newer, more specialized pattern than the other four mechanisms here — most teams still run draft and target together on the same replica — but it's an active area of 2026 inference-engine development as clusters look for ways to cut Time-To-First-Token without over-provisioning the largest GPUs for every request.

Inside a single GPU, continuous (iteration-level) batching — the technique behind vLLM and TensorRT-LLM's throughput gains — refills a finished sequence's batch slot with a new request at the next decoding step instead of waiting for the whole batch to complete. That maximizes GPU occupancy locally. The same signal is valuable one layer up: routers that monitor each replica's active batch-slot utilization can send new requests to whichever GPU has real spare decode capacity right now, rather than one that merely has the lowest connection count.

This matters because "least connections" and "batch slots actually free" frequently disagree — a replica can show few open connections while its batch is still saturated with long-running generations, or vice versa.

In production, these mechanisms aren't used in isolation — they're combined into a single scoring function per candidate GPU. A modern inference router (the pattern used by NVIDIA Dynamo, llm-d's Gateway API Inference Extension, and SGLang's router) typically blends: prefix/KV-cache hit probability, current cache memory pressure, queue depth, active decode-batch load, worker role (prefill vs. decode), LoRA-adapter availability if the deployment serves multiple fine-tunes, and request priority or SLA class — then routes to whichever replica scores best across all of them, not just one.