LLM Inference Engines (2026) | vLLM, SGLang, TensorRT-LLM, llama.cpp, Ollama, Dynamo, llm-d & Serving Frameworks

High-Throughput GPU Serving Engines

16 entries

Engines built for batching, KV-cache management and throughput.

01vLLMFast, memory-efficient LLM serving with PagedAttention, continuous batching and wide hardware support. 02vLLM on GitHubSource code and releases for vLLM. 03SGLangFast serving framework with RadixAttention prefix caching and a structured generation language. 04SGLang on GitHubSource code and releases for SGLang. 05TensorRT-LLMNVIDIA's optimised library for LLM inference on NVIDIA GPUs. 06NVIDIA DynamoDistributed inference framework with disaggregated prefill and decode, working with vLLM, SGLang and TensorRT-LLM. 07NVIDIA Dynamo-TritonFormerly Triton Inference Server; multi-framework model serving. 08LMDeployToolkit for compressing, deploying and serving LLMs, with the TurboMind engine. 09LightLLMLightweight Python LLM inference and serving framework. 10DeepSpeed-MIILow-latency, high-throughput inference from DeepSpeed. 11LoRAXMulti-LoRA inference server that serves many fine-tuned adapters on one GPU. 12Hugging Face Text Generation Inference (archived)TGI; the repository is now archived, so prefer vLLM or SGLang for new deployments. 13Text Embeddings InferenceFast inference for embedding and reranking models. 14XinferenceUnified serving for language, embedding and multimodal models. 15FastChatPlatform for training, serving and evaluating chat models. 16ExLlamaV2Fast inference library for quantized models on consumer GPUs.

Local, Desktop & Edge Runtimes

19 entries

Run models on laptops, phones and devices.

17llama.cppC/C++ LLM inference with GGUF models on CPUs, Apple silicon and GPUs. 18OllamaSimple local runner for open models. 19LM StudioDesktop app to download and run local models. 20llamafileSingle-file executables that bundle a model and llama.cpp. 21LocalAISelf-hosted, OpenAI-compatible API for local models. 22RamaLamaRun AI models in containers with a Podman-style workflow. 23mistral.rsFast LLM inference engine written in Rust. 24CandleMinimalist machine learning framework in Rust. 25MLXApple's array framework for machine learning on Apple silicon. 26mlx-lmRun and fine-tune LLMs with MLX. 27MLC LLMUniversal LLM deployment across GPUs, phones and browsers. 28BitNetInference framework for 1-bit LLMs. 29ExecuTorchPyTorch runtime for on-device inference. 30LiteRTGoogle's on-device runtime, formerly TensorFlow Lite. 31ONNX Runtime GenAIGenerative AI loop for ONNX Runtime. 32ONNX RuntimeCross-platform inference accelerator. 33OpenVINOIntel toolkit for optimised inference on CPUs, GPUs and NPUs. 34OpenVINO GenAIGenerative AI pipelines on OpenVINO. 35IPEX-LLM (archived)Intel's LLM acceleration library; the repository is archived.

Kernels, Caching & Core Techniques

9 entries

The building blocks inside the engines.

36FlashAttentionFast, memory-efficient exact attention kernels. 37FlashInferKernel library for LLM serving: attention, sampling and more. 38LMCacheKV-cache layer that shares and offloads cache across engines. 39Efficient Memory Management for LLM Serving with PagedAttentionThe vLLM paper. 40SGLang: Efficient Execution of Structured Language Model ProgramsThe SGLang paper. 41FlashAttention paperFast and memory-efficient exact attention with IO-awareness. 42Fast Inference from Transformers via Speculative DecodingSpeculative decoding paper. 43Optimizing LLMs for Speed and Memory (Hugging Face)Guide to quantization, Flash Attention and architectural tricks. 44LLM Training and Inferencing Optimization TechniquesOptimization techniques on this site.

Kubernetes & Distributed Serving Stacks

9 entries

Run engines at scale.

45llm-dKubernetes-native distributed LLM inference stack. 46vLLM Production StackReference stack for deploying vLLM on Kubernetes. 47AIBrixCloud-native building blocks for scalable vLLM serving. 48KServeKubernetes-native model serving. 49Ray ServeScalable model serving library. 50BentoMLFramework and platform for model serving. 51OpenLLMRun open LLMs with an OpenAI-compatible API. 52vLLM AscendvLLM plugin for Huawei Ascend NPUs. 53AMD ROCmAMD GPU software stack used by vLLM and SGLang.

Benchmarks & Load Testing

6 entries

Measure throughput and latency.

54MLPerf Inference: DatacenterIndustry benchmark for datacenter inference. 55MLPerf Inference: EdgeIndustry benchmark for edge inference. 56Artificial AnalysisIndependent model and API provider benchmarks. 57GuideLLMBenchmark LLM deployments under realistic load. 58LLMPerfBenchmarking tool for LLM APIs. 59genai-benchToken-level benchmarking for LLM serving.

Managed Inference Platforms

13 entries

Hosted engines as a service.

60BasetenInference platform for production models. 61Fireworks AIFast inference for open models. 62Together AIInference and training cloud for open models. 63ModalServerless compute for AI inference. 64AnyscaleRay-based platform. 65GroqLPU-based inference. 66CerebrasWafer-scale inference. 67SambaNovaDataflow-based inference. 68Databricks Model ServingManaged model serving. 69Amazon BedrockManaged foundation model access. 70Amazon SageMaker AI: DeployDeploy models on SageMaker. 71Google Cloud Vertex AIGoogle Cloud's model platform; the old Vertex AI address now redirects here. 72Microsoft FoundryFormerly Azure AI Foundry.

13 entries

Neighbouring pages on this site.

73LLM Inference PlatformsLLM inference platforms. 74Production AI Inference StackProduction inference stack. 75ML Model Serving ToolsModel serving tools. 76AI Model Quantization TechniquesQuantization techniques. 77LLM Model Formats ExplainedGGUF, safetensors and other formats. 78AI GatewaysAI and LLM gateways. 79LLMOpsLLMOps. 80Open-Source LLMsOpen-source LLMs. 81Small Language ModelsSmall language models. 82GPU Cloud ProvidersGPU clouds. 83AI Chip ProvidersAI chips. 84Edge AIEdge AI. 85Large Language ModelsLLM hub.

Frequently Asked Questions

What is an LLM inference engine?

Software that loads a trained language model and runs it efficiently, handling batching, KV-cache memory, scheduling, quantization and token generation. vLLM, SGLang and TensorRT-LLM are leading examples.

vLLM or SGLang?

Both deliver high throughput on GPUs. vLLM has the broadest model and hardware support and ecosystem; SGLang is strong for prefix-heavy and structured-output workloads and agentic pipelines. Benchmark both on your own traffic.

Which engine should I use locally?

llama.cpp or Ollama for CPUs and laptops, MLX on Apple silicon, and MLC LLM, ExecuTorch or LiteRT for phones and embedded devices.

How are engines scaled in production?

On Kubernetes with stacks such as llm-d, the vLLM Production Stack, AIBrix or KServe, often with disaggregated prefill and decode and shared KV-cache layers like LMCache.

How current is this list?

Links were checked in October 2026. Archived projects are labelled; sites that block automated checks are kept and may need a manual click.

LLM Inference Engines: vLLM, SGLang, TensorRT-LLM, llama.cpp & More