vLLM vs. SGLang vs. TensorRT-LLM vs. llama.cpp vs. Ollama: choosing an inference engine
Every one of these projects runs the same basic operation — turning a trained LLM's weights into fast, useful text generation. But they optimize for completely different things: raw multi-user throughput, KV-cache reuse across long shared prefixes, peak FLOPS on data-center silicon, or the ability to run a 7B model on a laptop with no GPU at all. Picking the wrong one doesn't just cost performance — it can mean rebuilding your serving stack later.
The five engines, profiled
Same job — turn weights into tokens — five very different architectures.
Core mechanism: PagedAttention manages the KV cache like virtual memory — paging it into non-contiguous blocks so memory is used almost fully instead of being wasted on padding. That, plus continuous batching, is what lets it serve many concurrent requests efficiently.
Hardware
- NVIDIA (primary)
- AMD ROCm, CPU, TPU, Gaudi, Ascend (secondary)
Ideal use case
- Small-team to mid-scale production serving
- The default choice for a self-hosted LLM API
Strengths
- Broad backend support
- Tensor/pipeline/data/expert parallelism
Watch out for
- Dependency drift — needs precise CUDA/PyTorch pinning
Core mechanism: RadixAttention organizes the KV cache in a radix tree, so requests that share a long common prefix — a system prompt, a RAG context block, a multi-turn chat history — reuse cached computation instead of recomputing it.
Hardware
- NVIDIA, AMD MI300/MI355
- Intel Xeon, Google TPU, Ascend NPU
Ideal use case
- RAG pipelines and multi-turn chat with repeated context
- Structured/constrained output generation
Strengths
- Large speedups specifically in high prefix-reuse workloads
Watch out for
- The advantage is workload-specific, not a universal speedup
Core mechanism: Compiles models into optimized NVIDIA runtime engines — kernel fusion, in-flight batching, and quantization tuned specifically to the target GPU — extracting close to peak hardware FLOPS.
Hardware
- NVIDIA-exclusive: H100, H200, L4, RTX, Jetson AGX Orin
Ideal use case
- Large-scale production on NVIDIA data-center GPUs
- Typically paired with Triton Inference Server
Strengths
- Maximum throughput per GPU on NVIDIA hardware
Watch out for
- NVIDIA-only; no native app-facing API — needs a Triton wrapper; steeper MLOps overhead
Core mechanism: A C/C++ inference runtime built around GGUF quantized model files, single-stream focused, and portable across an unusually wide range of hardware backends.
Hardware
- CUDA, HIP/ROCm, Metal, Vulkan, OpenCL
- AVX/AVX-512 CPU, RISC-V, Intel GPU
Ideal use case
- Single-user local inference; edge and non-NVIDIA hardware
- CPU-plus-GPU hybrid workloads
Strengths
- Runs almost anywhere; very CPU-efficient
Watch out for
- Single-stream focus — not built for concurrent multi-user serving
Core mechanism: A friendly wrapper around llama.cpp that handles model pulling, automatic quantization selection, and a simple CLI/API — optimized for a smooth "one command and it runs" experience.
Hardware
- NVIDIA CUDA, AMD ROCm, Apple Silicon Metal, CPU fallback
Ideal use case
- Local development, prototyping, single-user laptops/workstations
Strengths
- Deliberately short install path; broad hardware coverage out of the box
Watch out for
- Limited multi-GPU support; not designed for concurrent batched serving
Side by side
| Dimension | vLLM | SGLang | TensorRT-LLM | llama.cpp | Ollama |
|---|---|---|---|---|---|
| Core mechanism | PagedAttention | RadixAttention | Compiled kernels | GGUF runtime | llama.cpp wrapper |
| Primary hardware | NVIDIA (+ multi-backend) | NVIDIA, AMD, TPU, NPU | NVIDIA only | Almost anything | NVIDIA, AMD, Apple, CPU |
| Best for | Multi-user production serving | Prefix-heavy RAG / chat | Max throughput at scale | Edge / non-NVIDIA / portability | Local dev & prototyping |
| Concurrency model | Continuous batching | Continuous batching + cache reuse | In-flight batching | Single-stream focused | Single-stream focused |
| Setup complexity | Moderate | Moderate | High | Low | Very low |
| App-facing API | Built-in OpenAI-compatible server | Built-in OpenAI-compatible server | Needs Triton wrapper | Built-in server | Built-in CLI/API |
Which one should you actually use?
The right engine tracks to your deployment shape more than to raw benchmark numbers.
Prototyping on a laptop
Ollama. Fastest path from "I have a model name" to a running local endpoint, with sane defaults.
Non-NVIDIA or edge hardware
llama.cpp. Broadest hardware coverage of any engine here, and efficient on CPU-only boxes.
Self-hosted production API
vLLM. The default serving choice for teams running their own OpenAI-compatible endpoint at moderate concurrency.
RAG or long multi-turn chat at scale
SGLang. RadixAttention's prefix-sharing pays off specifically when many requests reuse the same context.
Max throughput on data-center NVIDIA GPUs
TensorRT-LLM. Worth the setup overhead when you're optimizing cost-per-token at real scale.
Not sure yet
Start with vLLM or Ollama depending on whether you're prototyping or serving — both have the gentlest path to "it works," and you can graduate to SGLang or TensorRT-LLM once your workload pattern is clear.
How they relate to each other
These aren't five competitors picking from the same menu — they cluster into families. llama.cpp is the portable C/C++ foundation that Ollama (and several other local-inference tools) wrap for a friendlier developer experience. vLLM and SGLang are both Python-native, cluster-oriented servers built for concurrent production traffic, differentiated mainly by their KV-cache strategy — pure paging versus prefix-aware reuse. TensorRT-LLM sits apart as NVIDIA's own compiled runtime, trading portability entirely for peak throughput on its own hardware.
There's no single "fastest" inference engine — only the fastest engine for your hardware, your concurrency pattern, and how much of your traffic shares a prefix. — The recurring lesson of every inference-engine benchmark
In practice, many teams end up running more than one: llama.cpp or Ollama on a developer's laptop during prototyping, vLLM or SGLang for the production API once the model and traffic pattern are settled, and TensorRT-LLM as the final throughput optimization once cost-per-token on NVIDIA infrastructure starts to matter.