Enterprise & Governance-First AI Gateways
9 Platforms-
TrueFoundry AI Gateway
Enterprise-grade gateway managing 1,600+ models with unified governance, real-time observability, guardrails, MCP integration, and routing/fallback across 250+ LLM providers.
Visit → -
SS&C AI Gateway
Vertically integrated, governance-first platform for regulated industries like financial services, offering secure LLM chat, pre-built agents, and a developer toolkit with built-in guardrails.
Visit → -
Kong AI Gateway
Enterprise AI connectivity platform governing LLM, MCP, and agent-to-agent traffic with centralized security, semantic caching, quota management, and multi-LLM routing in one gateway.
Visit → -
Glean AI Gateway
Enterprise control plane governing access to 40+ frontier and open models plus MCP tool traffic, with prompt-injection protection, restricted-topic controls, and Enterprise Graph context tying usage to apps, data, and people.
Visit → -
Databricks Unity Gateway
Databricks’ AI gateway for models, agents and MCP servers, with centralized governance, security, observability and cost management on the Databricks platform.
Visit → -
Azure API Management AI Gateway
Policies and features in Azure API Management to manage, secure, scale, monitor and govern LLM deployments, AI APIs and MCP servers.
Visit → -
F5 AI Gateway
Control plane for enterprise AI from F5, applying guardrails, security and cost controls across models, agents and MCP tools.
Visit → -
Lunar.dev
Self-hosted control plane for MCP, LLM and API traffic, with runtime governance and a single audit trail across AI interactions.
Visit → -
Traefik AI Gateway
Traefik Labs' gateway layer that, per its own site, turns any AI endpoint into a managed API with access control and cost and policy management.
Visit →
Open-Source & Self-Hosted AI Gateways
7 Platforms-
Agent Router (formerly Envoy AI Gateway)
Open-source control plane for agentic traffic built on Envoy, renamed from Envoy AI Gateway and now an Agentic AI Foundation project.
Visit → -
agentgateway
Open-source gateway that handles LLM, MCP, A2A and HTTP traffic in one data plane.
Visit → -
Apache APISIX AI Gateway
Apache APISIX used as an open-source LLM gateway, with multi-provider proxying, load balancing, retries, token limits, prompt controls and observability.
Visit → -
Higress
Open-source AI gateway, a CNCF Sandbox project built on Istio and Envoy, giving unified access, traffic governance, security and observability for LLM APIs, agents and MCP servers.
Visit → -
Plano
AI-native proxy and data plane for agentic apps, covering agent orchestration, LLM routing, guardrails and observability.
Visit → -
KrakenD
Stateless, high-performance open-source API gateway extended with AI capabilities — unified LLM interfaces, prompt guardrails, token/cost control, and a built-in MCP server, handling 80K+ requests/second.
Visit → -
kgateway
Open-source (Apache-2.0) Envoy-based gateway for Kubernetes that its repository describes as a cloud-native API gateway and AI gateway.
Visit →
MCP & Agent Gateways
3 Platforms-
IBM ContextForge
Open-source AI gateway, registry and proxy in front of MCP, A2A and REST or gRPC APIs, with one endpoint, centralized discovery, guardrails and plugins.
Visit → -
Docker MCP Gateway
Docker’s open-source gateway that runs and manages MCP servers in containers behind a single endpoint for AI clients.
Visit → -
Cequence AI Gateway
Agentic AI security and governance layer that connects agents to applications and data, enforcing least-privilege access and judging each agent action in context to prevent overreach.
Visit →
Open-Source & Self-Hosted LLM Gateways
5 Platforms-
LiteLLM
Open-source AI gateway and LLM proxy putting 140+ providers and 1,800+ models behind one OpenAI-compatible key, with spend tracking, smart routing, and self-hosting anywhere, even air-gapped.
Visit → -
Bifrost
Open-source (Apache 2.0), self-hostable AI gateway unifying 1,000+ models across 8+ providers behind a drop-in OpenAI/Anthropic-compatible SDK — with provider failover, budgeting, MCP gateway support, and built-in observability.
Visit → -
Otari (Mozilla.ai)
Open-source, OpenAI-compatible LLM gateway you run yourself, built on Mozilla.ai’s any-llm, with virtual keys, budgets and usage tracking across 40+ providers; a hosted option adds routing, observability and team management.
Visit → -
LLM Gateway
Open-source (AGPLv3), self-hostable unified API for 40+ LLM providers with real-time cost analytics, automatic failover, and drop-in OpenAI SDK compatibility; bring-your-own-keys tier is free.
Visit → -
MLflow AI Gateway
Gateway built into the open-source MLflow AI platform, exposing an OpenAI-compatible endpoint for any LLM provider with native tracing, evaluation, and MCP server access management, licensed Apache 2.0.
Visit →
Managed & Hosted LLM Gateways
5 Platforms-
OpenRouter
Unified interface for 400+ models across 70+ providers, letting developers find the best model and price for any prompt with automatic fallback and edge-optimized routing.
Visit → -
Cloudflare AI Gateway
Intelligent control plane for AI applications offering dynamic model routing, response caching, usage analytics, and unified billing across providers, built on Cloudflare's global network.
Visit → -
Vercel AI Gateway
One endpoint for hundreds of AI models from Vercel, without managing separate rate limits and API keys for each provider.
Visit → -
Requesty
OpenAI-compatible router for 600+ models, with smart routing, caching, failover, observability and EU data residency.
Visit → -
Portkey
LLM gateway with routing, guardrails, observability and governance, available as open-source and hosted. Palo Alto Networks completed its acquisition in May 2026.
Visit →
Secure & Compliance-Focused LLM Gateways
1 Platform-
SUPERWISE Sentinel
Commercial LLM gateway that redacts PII and secrets from every request inside your own network before the prompt reaches a public LLM, for teams with data-protection and compliance requirements.
Visit →
LLM Routing & Cost-Optimization Frameworks
2 Platforms-
RouteLLM
Open-source framework for serving and evaluating LLM routers, which send easy queries to cheaper models to cut costs without losing quality. The repository's last commit was in August 2024, so check its activity before adopting it.
Visit → -
vLLM Semantic Router
Programmable mixture-of-models router from the vLLM project that picks the right model for each request across heterogeneous LLM inference.
Visit →
What Makes an AI Gateway Distinct: 12 Core Capabilities
Field guideA traditional API gateway routes requests. An AI gateway also has to meter tokens, absorb a provider outage without anyone noticing, and decide in milliseconds whether a prompt is trying to leak a secret. Here is what sits inside one.
Why LLM traffic doesn't fit a regular API gateway
Four properties of LLM traffic don't show up in typical REST or gRPC workloads, and each one forces a different design decision.
Streaming by default
Responses arrive token-by-token over minutes, not milliseconds — the gateway has to speak SSE and WebSockets natively, not just proxy a request/response cycle.
Token economics
Cost and load aren't request counts — a single call can burn anywhere from a few hundred to a few hundred thousand tokens, so metering has to happen inside the payload, not at the connection.
Non-deterministic output
The same input can produce a different — sometimes unsafe — output on every call, which regular schema validation was never built to catch.
Provider heterogeneity
Dozens of providers, each with its own API shape, rate limits, and failure modes, all need to look like one consistent endpoint to the calling application.
Traffic Control
The entry point: who gets in, what they're allowed to send, and how much of it.
Centralized Access
Every application talks to one endpoint instead of wiring direct SDK calls to OpenAI, Anthropic, Google, and whichever provider gets added next. That single choke point is what makes every other capability on this list possible — you can't enforce a budget, redact a payload, or fail over to a backup model on traffic the gateway never sees. A single production gateway routing to well over a hundred models across providers is now a fairly ordinary deployment, not an edge case.
Authentication & Authorization
Every request is verified before it's allowed anywhere near a model, and role-based policy decides not just who can call the gateway but which models, which data classifications, and which spend tier they're allowed to touch. In agentic systems this extends further: an agent acting on a user's behalf needs a scoped, delegated identity rather than a shared service credential, so that one compromised agent can't act with another user's full permissions.
Request Validation
Schema, payload size, and content get checked before anything reaches a model — malformed JSON, an oversized context dump, or a request missing required fields is rejected at the edge instead of burning inference cost on something that was always going to fail.
Rate Limiting
This is where AI gateways diverge sharply from API gateways: limiting by request count alone doesn't protect anything, because two requests from the same user can differ in cost by three orders of magnitude depending on context length. Production rate limiting has to combine request-count, token-count, and concurrency limits, scoped per user, per application, and per API key, so one runaway script can't consume an entire team's provider quota.
Economics & Performance
Where the spend actually goes, and how much of it can be avoided.
Cost Controls
Budgets, quotas, and usage limits are enforced before spend gets out of hand — not read off a bill afterward. That means real-time token-level tracking broken down by user, team, model, provider, and sometimes geography, so a finance owner can see exactly which application, and which prompt pattern, is driving the number up.
Intelligent Routing
Requests get sent to the model actually suited to them — by cost, latency, capability, or plain availability — instead of hard-coding one model into application code. Strategies range from simple round-robin and least-busy routing to cost-weighted selection, where a provider charging several times more for the same capability gets proportionally less traffic. The payoff is also resilience: routing logic is what lets a gateway silently redirect traffic away from a provider that's degraded, before anyone downstream notices.
Response Caching
Frequent responses get reused rather than regenerated. Exact-match caching handles identical requests; semantic caching goes further, recognizing that "What's the capital of France?" and "Tell me France's capital city" are the same question and serving the cached answer for both. The latency difference is stark — a cache hit returns in single-digit milliseconds against several seconds for a live generation — and well-tuned semantic caches routinely cut both latency and inference spend by a meaningful double-digit percentage.
Reliability & Visibility
What happens when something breaks, and how anyone finds out.
Observability
Every request, error, latency figure, usage number, and success rate gets tracked centrally — not scattered across each application's own logging. This is what turns "the model felt slow today" into an actual root cause: a specific provider, a specific model version, a specific spike in context length, visible in one trace.
Retries & Fallbacks
When a provider or model fails, the gateway recovers instead of the failure reaching the user. That typically means automatic retries with exponential backoff, a circuit breaker that stops sending traffic to a provider once its failure rate crosses a threshold, and layered fallback rules — a general fallback for timeouts, a separate one for content-policy rejections, and another for context-window overruns, since each failure mode calls for a different next step.
Security & Change Management
Protecting what flows through the gateway, and evolving it without breaking anyone downstream.
Privacy & Redaction
Sensitive data gets masked before a prompt ever leaves the organization's boundary — names, account numbers, credentials, health information — so a third-party model provider never sees what it doesn't need to. This capability has expanded well beyond redaction alone: gateways increasingly run inline prompt-injection detection and output filtering too, since content-aware attacks are the risk category a conventional API gateway's security model was never built to catch.
Encryption
Communication in transit and data at rest — logs, cached responses, audit trails — are protected the same way any other sensitive enterprise data would be, including meeting data-residency requirements that dictate which region a request, and its cached response, is allowed to be processed or stored in.
Versioning
Model and API changes roll out without breaking applications already depending on the previous behavior. A provider deprecating a model, or a team wanting to test a newer version against a small slice of traffic, becomes a routing-configuration change at the gateway rather than a coordinated redeploy across every consuming application.
Where the AI gateway fits once agents enter the picture
An agent does not just call a model: it calls tools, queries data and sometimes calls other agents, and each needs its own governance. Enterprise designs increasingly describe a family of specialised gateways, tied together by identity and delegated permissions, observability and cost attribution, and governance and audit trail.
Unified model access — routing, caching, cost accounting, failover.
Controls which Model Context Protocol tools and servers an agent can discover and call — the piece that prevents ungoverned "shadow" tool access.
Governs agent runtime behavior itself — identity, execution limits, human-approval checkpoints.
Enforces access control and masking on the data an agent is allowed to query.
Makes sure a RAG pipeline respects document-level permissions, not just index-level access.
Inline inspection for prompt injection and data leakage across the whole request path.
The traditional layer, still handling the non-AI traffic these systems also depend on.
Three capabilities run across all of them rather than living in any single gateway: identity and delegated permissions, observability and cost attribution, and governance and audit trail.
Choosing a gateway: build vs. buy
The landscape spans open-source proxies you run yourself to fully managed platforms. The right choice depends more on team size and compliance posture than on any single feature checklist.
| Gateway | Model | Known for |
|---|---|---|
| LiteLLM | Open-source | Developer-first, OpenAI-compatible API in front of 100+ models; easy to self-host, needs hardening for regulated environments |
| Portkey | Commercial / SaaS | Prompt- and model-aware routing with strong token-level observability |
| Kong AI Gateway | Commercial / open-source | Traditional API gateway extended for AI; strong in Kubernetes and existing enterprise security stacks |
| Cloudflare AI Gateway | Managed, edge | Edge-cached responses; reports large latency reductions on cache hits |
| AWS Bedrock | Managed service | Serverless access to proprietary models; simplicity over long-run cost efficiency at scale |
| TrueFoundry | Commercial / SaaS | Enterprise control plane emphasizing lifecycle management and environment-based governance |
| Envoy AI Gateway | Open-source | Built on Envoy proxy; fits teams already standardized on Envoy/service-mesh infrastructure |
Teams rarely need all twelve capabilities on day one. Centralized access, authentication, and basic rate limiting are the floor — get those in place before a single application ships against a real model. Cost controls and retries/fallbacks are the next tier, worth adding the moment more than one provider is in play. Privacy, redaction, and fine-grained routing tend to arrive last, driven by compliance requirements and provider count rather than by raw traffic volume.
An AI gateway earns its name by handling the things a regular API gateway was never asked to: metering by token instead of by request, absorbing a model provider's bad day without anyone downstream noticing, and inspecting content that changes on every call instead of validating a fixed schema. As agentic systems add tool calls, data access, and multi-agent coordination on top of plain model calls, the AI gateway doesn't go away — it becomes the first, most heavily used link in a small chain of purpose-built control points that governs how an enterprise's AI systems are actually allowed to act.
Frequently Asked Questions
What is an AI gateway?
An AI gateway is a layer between applications and model providers. It gives every application one endpoint, and handles authentication, routing, rate limiting by tokens, cost controls, caching, observability, fallbacks and content security for LLM, MCP and agent traffic.
What is the difference between an AI gateway and an LLM gateway?
The terms overlap. LLM gateways focus on unifying model-provider APIs with routing, fallbacks, caching and budgets. AI gateways extend this to governing MCP tool calls and agent-to-agent traffic, with stronger security and policy. This page covers both.
How is an AI gateway different from an API gateway?
A traditional API gateway routes requests and validates fixed schemas. LLM traffic streams token by token, is priced by token rather than by request, varies in output and spans many providers, so the gateway has to meter tokens, fail over between providers and inspect content.
Should I self-host a gateway or use a managed one?
Managed gateways are quicker to adopt and need little engineering time. Self-hosting gives control over data residency and cost at scale but you operate and harden it. Team size and compliance requirements usually decide.
Which capabilities do I need first?
Centralised access, authentication and basic rate limiting are the floor. Add cost controls and retries or fallbacks once more than one provider is in use, and privacy, redaction and fine-grained routing as compliance needs grow.