A New Preview Layer for AI Access

Microsoft has released a public preview of a dedicated AI Gateway tier for Azure API Management, positioning it as a gateway resource organized around models, MCP servers and tools rather than conventional APIs.

The new tier is presented as a standalone experience, not simply another policy layer on top of existing gateways. Existing AI gateway capabilities in the classic and v2 tiers remain available.

What the Gateway Covers

The preview reflects a common enterprise pattern: teams often need to connect to multiple model providers at the same time. The gateway can publish models hosted on Foundry, including OpenAI, Anthropic and Mistral, as well as models from AWS Bedrock, Google Vertex AI and OpenAI directly.

For OpenAI-compatible providers, requests share a common endpoint path and are routed by exact matching on the model field, which means every published model needs a unique name. Anthropic is handled through a custom provider that passes through the Messages API.

Policies are configured through portal cards rather than XML expressions. They cover token and request limits, quotas, content safety and model failover. Provisioning takes about one minute, with no need to plan scale units in advance. Telemetry is exported as OpenTelemetry token metrics to targets such as Application Insights, Datadog and Grafana, while the resource runs in the customer’s own Azure subscription and Entra tenant.

For tools, the gateway can federate three backend types: remote MCP servers connected by URL, OpenAPI specifications, and built-in connectors for more than one thousand SaaS applications. Each backend operation becomes a tool, and teams can choose no authentication, API keys, OAuth 2.0 or managed identity per backend.

Governance Benefits and Open Questions

Microsoft’s intended operating model separates centralized governance from application-team self-service. Platform teams connect and publish approved models and tools, while application teams use those assets in a test console and build on top of them without routing every change through a central team.

The response from architects and platform engineers has been broadly positive. Paolo Perrone, who writes the AI Engineer newsletter, highlighted cost governance as an underrated part of the release: many teams only add rate limits and spending controls after incidents, while a gateway offers one control plane instead of patches in every application.

Still, important lifecycle questions remain. Enterprise AI systems architect Adolph White Jr. asked what happens when an agent run produces useful output but does not complete normally. Should those outputs be preserved for audit review, or should the gateway fail over and retry? That distinction separates governing AI traffic from governing the full lifecycle of an AI-driven task. Microsoft’s announcement does not clarify whether control over what agent outputs may change belongs in the gateway or in a higher orchestration layer.

Permission Scope and Preview Caveats

The sharpest design concern is key scope. Runtime access keys apply to the entire gateway, meaning a key can access every model and every tool published on that gateway. Microsoft recommends one key per application, but if such a key leaks, the blast radius is the whole gateway rather than an individual product. Teams that currently use APIM subscriptions to restrict consumers to a defined API set will not find the same boundary here.

The preview status also matters. Availability is best-effort, with no SLA. APIs, telemetry, limits, regions and pricing may change before general availability. Preview quotas limit models, tools, runtime keys and throughput, but the specific limits have not been published. Pricing will be announced later in the preview, making the cost-governance argument important but still incomplete.

There is also confusion about coexistence. Some users responding to the announcement described features such as caching, content safety, logging and token limits that already exist in current APIM AI gateway capabilities, not necessarily in the newly previewed tier. Microsoft’s documentation describes AI gateway capabilities as an extension of API Management, while this new tier is positioned as a separate experience. Organizations that have built on Premium or Standard v2 do not yet have public guidance on whether those investments will be inherited, run in parallel or migrate.

Outlook

The AI Gateway tier is available in East US 2 and Sweden Central, free during preview, with tutorials for publishing governed Foundry models and observing token rate limits.

The direction is clear: enterprise AI infrastructure is moving from single-model integration toward centralized governance of models, tools, agents and telemetry. But Microsoft still needs to clarify migration paths, coexistence with existing APIM deployments and finer-grained permission boundaries. If those issues are addressed, the new gateway could become a practical control plane for Azure-based AI applications; if not, it may first become another layer that platform teams must govern carefully.