01Managed inference platforms
Together AI
Cloud platform for running open-source models through serverless and dedicated endpoints, with fine-tuning and GPU clusters.
Best for: Open models behind a simple API.
Visit Together AI →AI Inference Platforms & Tools
Managed inference APIs, custom-silicon clouds, hyperscaler platforms, GPU clouds, inference engines and serving frameworks for running AI models in production.
35platforms and tools
6Categories
5FAQs answered
02Managed inference platforms
Fireworks AI
Inference platform for open and custom models, focused on speed, throughput and production reliability.
Best for: Fast serving of open models.
Visit Fireworks AI →03Managed inference platforms
Baseten
Inference platform for deploying and scaling models in production, including custom and open-source models.
Best for: Production model serving with autoscaling.
Visit Baseten →04Managed inference platforms
DeepInfra
Pay-per-use API for running many open models without managing servers.
Best for: Low-friction access to open models.
Visit DeepInfra →05Managed inference platforms
Replicate
API and model library for running open-source and community models, with simple deployment of custom models.
Best for: Trying and shipping models quickly.
Visit Replicate →06Managed inference platforms
fal
Generative media platform for fast inference of image, video and audio models.
Best for: Generative media workloads.
Visit fal →07Managed inference platforms
Hugging Face Inference Endpoints
Managed service for deploying models from the Hugging Face Hub onto dedicated infrastructure.
Best for: Deploying Hub models without running servers.
Visit Hugging Face Inference Endpoints →08Managed inference platforms
OpenRouter
Unified API and marketplace that routes requests to many model providers with one account.
Best for: Switching providers behind one API.
Visit OpenRouter →09Custom-silicon inference clouds
Groq
Inference cloud built on Groq's own LPU chips, aimed at very fast token generation.
Best for: Latency-sensitive applications.
Visit Groq →10Custom-silicon inference clouds
Cerebras
AI company running inference on its wafer-scale processors, offered through a cloud API.
Best for: Very high output speed.
Visit Cerebras →11Custom-silicon inference clouds
SambaNova
AI platform built on its own reconfigurable dataflow chips, offering cloud inference and on-premises systems.
Best for: Enterprise inference on dedicated hardware.
Visit SambaNova →12Hyperscaler & edge platforms
Amazon Bedrock
AWS service for accessing foundation models from several providers through one API, with security and governance features.
Best for: Managed models inside AWS.
Visit Amazon Bedrock →13Hyperscaler & edge platforms
Microsoft Foundry models
Microsoft's catalogue and hosting for foundation models within Azure's AI platform.
Best for: Models within Azure.
Visit Microsoft Foundry models →14Hyperscaler & edge platforms
Google Gemini Enterprise Agent Platform
Google Cloud's AI platform for building and deploying models and agents, formerly known as Vertex AI.
Best for: Models and agents on Google Cloud.
Visit Google Gemini Enterprise Agent Platform →15Hyperscaler & edge platforms
Cloudflare Workers AI
Serverless inference on Cloudflare's global network, with a catalogue of open models.
Best for: Low-latency inference close to users.
Visit Cloudflare Workers AI →16GPU clouds & serverless compute
Modal
Serverless compute platform for running code and models on GPUs with fast cold starts.
Best for: Python-first serverless GPU workloads.
Visit Modal →17GPU clouds & serverless compute
RunPod
GPU cloud with pods and serverless endpoints for running and scaling AI workloads.
Best for: Affordable GPU endpoints.
Visit RunPod →18GPU clouds & serverless compute
CoreWeave
Specialised AI cloud providing large-scale GPU infrastructure for training and inference.
Best for: Large GPU capacity.
Visit CoreWeave →19GPU clouds & serverless compute
Lambda Inference
Lambda's inference API and GPU cloud for serving open models.
Best for: GPU cloud plus managed inference.
Visit Lambda Inference →20GPU clouds & serverless compute
Nebius
AI cloud with GPU infrastructure and managed inference services.
Best for: European AI cloud capacity.
Visit Nebius →21Inference engines & runtimes
vLLM
High-throughput open-source engine for serving large language models, known for PagedAttention and continuous batching.
Best for: The common default for self-hosted LLM serving.
Visit vLLM →22Inference engines & runtimes
SGLang
Open-source serving framework with a fast runtime for large language and multimodal models.
Best for: High-performance serving and structured generation.
Visit SGLang →23Inference engines & runtimes
TensorRT-LLM
NVIDIA's open-source library for optimised LLM inference on NVIDIA GPUs.
Best for: Maximum performance on NVIDIA hardware.
Visit TensorRT-LLM →24Inference engines & runtimes
llama.cpp
C/C++ inference engine that runs quantised models on CPUs and a wide range of GPUs.
Best for: Local and edge inference.
Visit llama.cpp →25Inference engines & runtimes
Ollama
Tool for downloading and running open models locally with a simple command line and API.
Best for: Easy local model use.
Visit Ollama →26Inference engines & runtimes
Text Generation Inference
Hugging Face's inference server for large language models. The project is now in maintenance mode, so check alternatives for new work.
Best for: Existing deployments that already use it.
Visit Text Generation Inference →27Inference engines & runtimes
ONNX Runtime
Cross-platform runtime for running ONNX models on many hardware targets.
Best for: Portable inference across devices.
Visit ONNX Runtime →28Serving frameworks & orchestration
NVIDIA Dynamo
NVIDIA's framework for distributed, disaggregated inference of generative models across many GPUs.
Best for: Large-scale multi-node serving.
Visit NVIDIA Dynamo →29Serving frameworks & orchestration
llm-d
Kubernetes-native project for distributed LLM inference with cache-aware routing and disaggregated serving.
Best for: Cluster-level LLM serving on Kubernetes.
Visit llm-d →30Serving frameworks & orchestration
NVIDIA Dynamo-Triton
NVIDIA's inference server, formerly Triton Inference Server, for serving models from many frameworks.
Best for: Multi-framework model serving.
Visit NVIDIA Dynamo-Triton →31Serving frameworks & orchestration
KServe
Kubernetes-based standard for serving machine learning and generative models, with autoscaling.
Best for: Model serving on Kubernetes.
Visit KServe →32Serving frameworks & orchestration
Ray Serve
Scalable model-serving library built on Ray, for composing models and business logic in Python.
Best for: Python model pipelines at scale.
Visit Ray Serve →33Serving frameworks & orchestration
BentoML
Framework and platform for packaging models as APIs and deploying them to production.
Best for: Packaging and shipping model services.
Visit BentoML →34Serving frameworks & orchestration
NVIDIA NIM
NVIDIA's prebuilt inference microservices that package optimised models for deployment on NVIDIA GPUs.
Best for: Quick, optimised deployments on NVIDIA GPUs.
Visit NVIDIA NIM →35Serving frameworks & orchestration
LMCache
Open-source KV-cache layer that reuses and shares computed attention state across requests and servers.
Best for: Cutting repeated prefill work.
Visit LMCache →Choosing the right approach
Match your requirement to a starting point, then compare vendors.
| Try open models fast, pay per use | A managed platform such as Together AI, Fireworks, DeepInfra or Baseten |
|---|---|
| Lowest latency per token | A custom-silicon cloud such as Groq, Cerebras or SambaNova |
| Stay within your cloud and governance | Bedrock, Microsoft Foundry or Google Cloud's platform |
| Own your GPUs and costs | A GPU cloud plus an engine such as vLLM or SGLang |
| Run on a laptop or edge device | llama.cpp, Ollama or ONNX Runtime |
| Scale across a GPU cluster | Dynamo, llm-d, KServe or Ray Serve |
Related pages on PeterIndia.net
Neighbouring directories and hubs.
LLM Inference Platforms and ToolsLLM API providers and hosts.The Production AI Inference StackDistributed serving explained.Platforms for Running LLMs LocallyPrivate, on-device models.GPU Cloud ProvidersWhere to rent GPUs.AI Chip ProvidersHardware behind inference.AI GatewaysRoute and govern model traffic.LLM Inference OptimizationMake serving faster and cheaper.LLM HubAll LLM resources in one place.
Frequently asked questions
Quick answers.
What is AI inference?
Inference is running a trained model to produce outputs, such as answering a prompt or classifying an image. It is the production phase that follows training, and it is where most ongoing cost and latency arise.
Should I use a managed platform or host my own?
Managed platforms are faster to start and handle scaling. Self-hosting with an engine such as vLLM gives control over data, cost and tuning but you operate the stack. Many teams start managed and move selected workloads in-house.
What is the difference between an engine and a platform?
An inference engine runs the model efficiently on hardware. A platform packages engines with GPUs, autoscaling, APIs and billing so you do not run the infrastructure yourself.
What affects inference cost and speed?
Model size, quantisation, batching, caching, context length, hardware and traffic pattern all matter. Measure time to first token, tokens per second and cost per million tokens on your own prompts.
How do I avoid vendor lock-in?
Prefer OpenAI-compatible APIs, keep prompts and evaluation sets portable, and consider a gateway or router so you can switch providers.