LLM Inference Platforms and Tools logopeterindia.netLLM Inference Platforms
Home / LLM Hub / LLM Inference Platforms and Tools
LLM infrastructure · 2026

LLM Inference Platforms & Tools

LLM API providers, open-model inference hosts, fast-chip and GPU clouds, hyperscaler platforms, routers, self-hosted serving frameworks and local desktop apps, plus a side-by-side comparison of the vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama inference engines.

55platforms and tools
8Categories
8FAQs answered
01Model-maker APIs

OpenAI API

OpenAI's developer platform for calling its GPT and reasoning models, with chat, tools, audio and image features.

Best for: Access to OpenAI's own models.

Visit OpenAI API →
02Model-maker APIs

Anthropic Claude API

Anthropic's API for the Claude model family, with tool use, long context and vision.

Best for: Claude models with strong coding and agent support.

Visit Anthropic Claude API →
03Model-maker APIs

Google Gemini API

Google's API for the Gemini models, with multimodal input, long context and tool use.

Best for: Multimodal and long-context workloads.

Visit Google Gemini API →
04Model-maker APIs

Mistral AI API

Mistral's API for its proprietary and open-weight models, with function calling and embeddings.

Best for: European provider with open and hosted models.

Visit Mistral AI API →
05Model-maker APIs

Cohere API

Cohere's API for its Command, embedding and rerank models, aimed at enterprise search and generation.

Best for: Enterprise RAG and retrieval.

Visit Cohere API →
06Model-maker APIs

xAI API

API for xAI's Grok models.

Best for: Access to Grok models.

Visit xAI API →
07Model-maker APIs

DeepSeek API

DeepSeek's API for its chat and reasoning models, using an OpenAI-compatible format.

Best for: Low-cost open-lab models behind a simple API.

Visit DeepSeek API →
08Model-maker APIs

Kimi Open Platform

Moonshot AI's developer platform for the Kimi models.

Best for: Long-context and agentic Kimi models.

Visit Kimi Open Platform →
09Model-maker APIs

Alibaba Cloud Model Studio

Alibaba Cloud's platform for using Qwen and other models through APIs.

Best for: Qwen models on Alibaba Cloud.

Visit Alibaba Cloud Model Studio →
10Model-maker APIs

Perplexity API

Perplexity's API for search-grounded answers and its models.

Best for: Answers that cite web sources.

Visit Perplexity API →
11Open-model inference hosts

DigitalOcean serverless inference

DigitalOcean's single endpoint, compatible with OpenAI and Anthropic APIs, for running production LLM workloads.

Best for: Teams already on DigitalOcean.

Visit DigitalOcean serverless inference →
12Open-model inference hosts

Together AI

Runs inference for open-weight models and supports LoRA adapters, full fine-tuning and DPO tuning.

Best for: Open models plus customisation.

Visit Together AI →
13Open-model inference hosts

Fireworks AI

Serves open-weight models and supports full-parameter fine-tuning on very large models.

Best for: Fast serving and fine-tuning of open models.

Visit Fireworks AI →
14Open-model inference hosts

Nebius

Managed inference API with a catalogue of dozens of models and compliance certifications.

Best for: Open models with enterprise compliance needs.

Visit Nebius →
15Open-model inference hosts

SiliconFlow

Platform with OpenAI-compatible endpoints for running a large catalogue of language and multimodal models.

Best for: Wide model choice in one place.

Visit SiliconFlow →
16Open-model inference hosts

Baseten

Inference platform that serves open-source, custom and fine-tuned models.

Best for: Custom models in production.

Visit Baseten →
17Open-model inference hosts

DeepInfra

Pay-per-use API for running many open models without managing servers.

Best for: Simple pay-as-you-go access.

Visit DeepInfra →
18Open-model inference hosts

Hugging Face Inference Providers

Hugging Face service that gives one interface to several inference providers for models on the Hub.

Best for: Hub models via a unified API.

Visit Hugging Face Inference Providers →
19Open-model inference hosts

NVIDIA API catalog

NVIDIA's catalogue for trying and calling optimised models, built on NVIDIA microservices.

Best for: Testing NVIDIA-optimised models.

Visit NVIDIA API catalog →
20Fast-chip inference clouds

Groq

Runs inference on its own custom LPU chips, aimed at low latency. Nvidia announced a non-exclusive licensing agreement for Groq's inference technology in December 2025, and Groq said GroqCloud would continue to operate.

Best for: Latency-sensitive chat and agents.

Visit Groq →
21Fast-chip inference clouds

Cerebras

Runs inference on its wafer-scale processors and offers it through a cloud API. Cerebras began trading on Nasdaq as CBRS on 14 May 2026, according to press reports.

Best for: Very high tokens per second.

Visit Cerebras →
22Fast-chip inference clouds

SambaNova

Cloud inference built on its own dataflow chips, also available as on-premises systems.

Best for: Enterprise inference on dedicated hardware.

Visit SambaNova →
23Cloud, GPU & serverless platforms

Amazon Bedrock

AWS service for using foundation models from several providers through one API, with security and governance features.

Best for: Models inside an AWS environment.

Visit Amazon Bedrock →
24Cloud, GPU & serverless platforms

Cloudflare Workers AI

Serverless inference on Cloudflare's network, with a catalogue of open models.

Best for: Edge-served inference close to users.

Visit Cloudflare Workers AI →
25Cloud, GPU & serverless platforms

Modal

Serverless compute platform for running GPU workloads, deployed from the command line.

Best for: Self-managed models without servers.

Visit Modal →
26Routers & aggregators

OpenRouter

Unified API layer over hundreds of models from many providers.

Best for: One account, many providers.

Visit OpenRouter →
27Routers & aggregators

LiteLLM

Open-source gateway and SDK that exposes many model providers through an OpenAI-style interface.

Best for: Provider switching and spend tracking.

Visit LiteLLM →
28Self-hosted serving & frameworks

vLLM

High-throughput open-source engine for serving LLMs, known for PagedAttention and continuous batching. v0.32.0 was released on 18 October 2026, according to GitHub.

Best for: The common default for self-hosted serving.

Visit vLLM →
29Self-hosted serving & frameworks

SGLang

Open-source serving framework with a fast runtime for language and multimodal models. v0.5.22 was released on 19 October 2026, according to GitHub.

Best for: High-performance serving and structured output.

Visit SGLang →
30Self-hosted serving & frameworks

TensorRT-LLM

NVIDIA's open-source library for optimised LLM inference on NVIDIA GPUs. The latest GitHub release listed is v1.2.1, from 20 April 2026, and its trtllm-serve command starts an OpenAI-compatible server.

Best for: Peak performance on NVIDIA hardware.

Visit TensorRT-LLM →
31Self-hosted serving & frameworks

llama.cpp

C/C++ engine that runs quantised models on CPUs and many GPUs.

Best for: Local and edge inference.

Visit llama.cpp →
32Self-hosted serving & frameworks

Ollama

Tool for downloading and running open models locally through a simple command line and API. v0.40.9 was released on 22 October 2026, according to GitHub.

Best for: Easy local use.

Visit Ollama →
33Self-hosted serving & frameworks

llm-d

Kubernetes-native project for distributed LLM inference with cache-aware routing and disaggregated serving. v1.0.0 was released on 20 October 2026, according to GitHub.

Best for: Cluster-scale serving on Kubernetes.

Visit llm-d →
34Self-hosted serving & frameworks

NVIDIA Dynamo

NVIDIA's open framework for distributed, disaggregated inference of generative models. v1.6.0 was released on 22 October 2026, according to GitHub.

Best for: Multi-node GPU serving.

Visit NVIDIA Dynamo →
35Open-model inference hosts

Replicate

API and model library for running open-source and community models, with simple deployment of custom models. Cloudflare completed its acquisition of Replicate on 1 December 2025, and Replicate continues as a distinct brand.

Best for: Trying and shipping models quickly.

Visit Replicate →
36Open-model inference hosts

fal

Generative media platform for fast inference of image, video and audio models.

Best for: Generative media workloads.

Visit fal →
37Open-model inference hosts

Hugging Face Inference Endpoints

Managed service for deploying models from the Hugging Face Hub onto dedicated infrastructure.

Best for: Deploying Hub models without running servers.

Visit Hugging Face Inference Endpoints →
38Cloud, GPU & serverless platforms

Microsoft Foundry models

Microsoft's catalogue and hosting for foundation models within Azure's AI platform.

Best for: Models within Azure.

Visit Microsoft Foundry models →
39Cloud, GPU & serverless platforms

Google Gemini Enterprise Agent Platform

Google Cloud's AI platform for building and deploying models and agents, formerly known as Vertex AI.

Best for: Models and agents on Google Cloud.

Visit Google Gemini Enterprise Agent Platform →
40Cloud, GPU & serverless platforms

RunPod

GPU cloud with pods and serverless endpoints for running and scaling AI workloads.

Best for: Affordable GPU endpoints.

Visit RunPod →
41Cloud, GPU & serverless platforms

CoreWeave

Specialised AI cloud providing large-scale GPU infrastructure for training and inference.

Best for: Large GPU capacity.

Visit CoreWeave →
42Cloud, GPU & serverless platforms

Lambda Inference

Lambda's inference API and GPU cloud for serving open models.

Best for: GPU cloud plus managed inference.

Visit Lambda Inference →
43Self-hosted serving & frameworks

Text Generation Inference

Hugging Face's inference server for large language models. The project was in maintenance mode and its GitHub repository was archived by its owner in March 2026, so check alternatives for new work.

Best for: Existing deployments that already use it.

Visit Text Generation Inference →
44Self-hosted serving & frameworks

ONNX Runtime

Cross-platform runtime for running ONNX models on many hardware targets.

Best for: Portable inference across devices.

Visit ONNX Runtime →
45Self-hosted serving & frameworks

NVIDIA Dynamo-Triton

NVIDIA's inference server, formerly Triton Inference Server, for serving models from many frameworks.

Best for: Multi-framework model serving.

Visit NVIDIA Dynamo-Triton →
46Self-hosted serving & frameworks

KServe

Kubernetes-based standard for serving machine learning and generative models, with autoscaling.

Best for: Model serving on Kubernetes.

Visit KServe →
47Self-hosted serving & frameworks

Ray Serve

Scalable model-serving library built on Ray, for composing models and business logic in Python.

Best for: Python model pipelines at scale.

Visit Ray Serve →
48Self-hosted serving & frameworks

BentoML

Framework and platform for packaging models as APIs and deploying them to production.

Best for: Packaging and shipping model services.

Visit BentoML →
49Self-hosted serving & frameworks

NVIDIA NIM

NVIDIA's prebuilt inference microservices that package optimised models for deployment on NVIDIA GPUs.

Best for: Quick, optimised deployments on NVIDIA GPUs.

Visit NVIDIA NIM →
50Self-hosted serving & frameworks

LMCache

Open-source KV-cache layer that reuses and shares computed attention state across requests and servers.

Best for: Cutting repeated prefill work.

Visit LMCache →
51Local & desktop LLM apps

LM Studio

Desktop application for running LLMs offline, with a graphical interface, model management and a built-in chat.

Best for: A friendly GUI for local models.

Visit LM Studio →
52Local & desktop LLM apps

KoboldCpp

Frontend for llama.cpp with a web interface, aimed at many models plus story writing and role-play.

Best for: Local creative writing and role-play.

Visit KoboldCpp →
53Local & desktop LLM apps

Jan

Open-source AI assistant that runs fully offline, with a desktop interface and support for multiple models.

Best for: A private desktop assistant.

Visit Jan →
54Local & desktop LLM apps

Msty

Local LLM application with a clean interface, easy model downloads and chat features.

Best for: Simple local chat with many models.

Visit Msty →
55Comparison

Artificial Analysis provider benchmarks

Independent comparison of API providers on price, output speed and latency for the same models.

Best for: Comparing providers before you commit.

Visit Artificial Analysis provider benchmarks →

Inference engines compared

vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama all turn model weights into tokens, but they optimise for different things. Adapted from the AI Infra Notes article by Pethuru (August 2026).

vLLM

Production multi-user serving

Core mechanism: PagedAttention manages the KV cache like virtual memory — paging it into non-contiguous blocks so memory is used almost fully instead of being wasted on padding. That, plus continuous batching, is what lets it serve many concurrent requests efficiently.

Hardware

  • NVIDIA (primary)
  • AMD ROCm, CPU, TPU, Gaudi, Ascend (secondary)

Ideal use case

  • Small-team to mid-scale production serving
  • The default choice for a self-hosted LLM API

Strengths

  • Broad backend support
  • Tensor/pipeline/data/expert parallelism

Watch out for

  • Dependency drift — needs precise CUDA/PyTorch pinning

SGLang

Prefix-heavy, structured generation

Core mechanism: RadixAttention organizes the KV cache in a radix tree, so requests that share a long common prefix — a system prompt, a RAG context block, a multi-turn chat history — reuse cached computation instead of recomputing it.

Hardware

  • NVIDIA, AMD MI300/MI355
  • Intel Xeon, Google TPU, Ascend NPU

Ideal use case

  • RAG pipelines and multi-turn chat with repeated context
  • Structured/constrained output generation

Strengths

  • Large speedups specifically in high prefix-reuse workloads

Watch out for

  • The advantage is workload-specific, not a universal speedup

TensorRT-LLM

Peak throughput on NVIDIA silicon

Core mechanism: Compiles models into optimized NVIDIA runtime engines — kernel fusion, in-flight batching, and quantization tuned specifically to the target GPU — extracting close to peak hardware FLOPS.

Hardware

  • NVIDIA-exclusive: H100, H200, L4, RTX, Jetson AGX Orin

Ideal use case

  • Large-scale production on NVIDIA data-center GPUs
  • Often paired with Dynamo-Triton, and also serves an OpenAI-compatible API through trtllm-serve

Strengths

  • Maximum throughput per GPU on NVIDIA hardware

Watch out for

  • NVIDIA-only; steeper MLOps overhead

llama.cpp

Maximum portability

Core mechanism: A C/C++ inference runtime built around GGUF quantized model files, single-stream focused, and portable across an unusually wide range of hardware backends.

Hardware

  • CUDA, HIP/ROCm, Metal, Vulkan, OpenCL
  • AVX/AVX-512 CPU, RISC-V, Intel GPU

Ideal use case

  • Single-user local inference; edge and non-NVIDIA hardware
  • CPU-plus-GPU hybrid workloads

Strengths

  • Runs almost anywhere; very CPU-efficient

Watch out for

  • Single-stream focus — not built for concurrent multi-user serving

Ollama

Local developer experience

Core mechanism: A friendly wrapper around llama.cpp that handles model pulling, automatic quantization selection, and a simple CLI/API — optimized for a smooth "one command and it runs" experience.

Hardware

  • NVIDIA CUDA, AMD ROCm, Apple Silicon Metal, CPU fallback

Ideal use case

  • Local development, prototyping, single-user laptops/workstations

Strengths

  • Deliberately short install path; broad hardware coverage out of the box

Watch out for

  • Limited multi-GPU support; not designed for concurrent batched serving

Side by side

DimensionvLLMSGLangTensorRT-LLMllama.cppOllama
Core mechanismPagedAttentionRadixAttentionCompiled kernelsGGUF runtimellama.cpp wrapper
Primary hardwareNVIDIA (+ multi-backend)NVIDIA, AMD, TPU, NPUNVIDIA onlyAlmost anythingNVIDIA, AMD, Apple, CPU
Best forMulti-user production servingPrefix-heavy RAG / chatMax throughput at scaleEdge / non-NVIDIA / portabilityLocal dev & prototyping
Concurrency modelContinuous batchingContinuous batching + cache reuseIn-flight batchingSingle-stream focusedSingle-stream focused
Setup complexityModerateModerateHighLowVery low
App-facing APIBuilt-in OpenAI-compatible serverBuilt-in OpenAI-compatible serverBuilt-in OpenAI-compatible server (trtllm-serve)Built-in serverBuilt-in CLI/API

Which engine should you use?

Prototyping on a laptopOllama. Fastest path from "I have a model name" to a running local endpoint, with sane defaults.
Non-NVIDIA or edge hardwarellama.cpp. Broadest hardware coverage of any engine here, and efficient on CPU-only boxes.
Self-hosted production APIvLLM. The default serving choice for teams running their own OpenAI-compatible endpoint at moderate concurrency.
RAG or long multi-turn chat at scaleSGLang. RadixAttention's prefix-sharing pays off specifically when many requests reuse the same context.
Max throughput on data-center NVIDIA GPUsTensorRT-LLM. Worth the setup overhead when you're optimizing cost-per-token at real scale.
Not sure yetStart with vLLM or Ollama depending on whether you're prototyping or serving — both have the gentlest path to "it works," and you can graduate to SGLang or TensorRT-LLM once your workload pattern is clear.

How they relate

These aren't five competitors picking from the same menu — they cluster into families. llama.cpp is the portable C/C++ foundation that Ollama (and several other local-inference tools) wrap for a friendlier developer experience. vLLM and SGLang are both Python-native, cluster-oriented servers built for concurrent production traffic, differentiated mainly by their KV-cache strategy — pure paging versus prefix-aware reuse. TensorRT-LLM sits apart as NVIDIA's own compiled runtime, trading portability entirely for peak throughput on its own hardware.

There's no single "fastest" inference engine — only the fastest engine for your hardware, your concurrency pattern, and how much of your traffic shares a prefix. — The recurring lesson of every inference-engine benchmark

In practice, many teams end up running more than one: llama.cpp or Ollama on a developer's laptop during prototyping, vLLM or SGLang for the production API once the model and traffic pattern are settled, and TensorRT-LLM as the final throughput optimization once cost-per-token on NVIDIA infrastructure starts to matter.

Choosing the right approach

Match your requirement to a starting point, then compare vendors.

Use a specific frontier modelThe model maker's own API, such as OpenAI, Anthropic or Google
Run open models without serversAn open-model host such as Together, Fireworks, Nebius or DeepInfra
Lowest latencyA fast-chip cloud such as Groq, Cerebras or SambaNova
Stay inside a cloud accountAmazon Bedrock or your cloud's own model service
Avoid lock-in to one providerA router or gateway such as OpenRouter or LiteLLM
Run models privately on a laptop or desktopA local app such as LM Studio, Jan or Msty, or Ollama and llama.cpp
Full control of data and costSelf-hosted serving with vLLM, SGLang or llm-d

Related pages on PeterIndia.net

Neighbouring directories and hubs.

Descriptions are summaries of each vendor's own positioning, checked against its website on 4 October 2026. Product names and offerings change often, so confirm details on the vendor site before deciding.