Businesses are using AI models from different providers, each with its own connections, login details, pricing and usage limits. Managing these separately might work for a few applications, but it gets much harder as more teams start using AI. An enterprise AI gateway gives businesses one place to manage those connections.
But having everything go through one system doesn't guarantee the business has control over what happens. PayPal’s research on its AI gateway from October 2026 reveals just how much these systems can do. For other firms, it’s a question of what their gateway really needs to control, when that control should happen and if they can rely on it when things go wrong.
What Does an Enterprise AI Gateway Actually Do?
A business AI gateway is an intermediary between a company’s applications and the AI models it is using. So if an application wants something from an AI model, it sends the request through the gateway, not directly to a provider. The gateway then checks the request and forwards it on to the right place.
It’s like a central hub for controlling access to AI. Businesses can decide which applications can use certain models, how requests are handled and how much AI each application can use. The gateway can also manage provider connections, login details and usage records, so developers don't have to build these controls into every application.
But there are limits to what a gateway can achieve. It’s not just automatically controlling everything that’s going on inside an AI app. For example, an AI agent could have its own memory or use other software tools to do a task. The gateway solely controls the requests that travel through it and the rules it was set up to enforce.
If the gateway goes down, every application relying on it could lose access to AI, even when the providers are still working.
Which Controls Must Run Before a Model Request?
The gateway needs to check requests before they reach an AI provider. If an application isn't allowed to use a model or has reached its spending limit, the gateway must stop the request before it's processed. Finding out afterwards is too late.
Authentication, permissions and provider credentials
First, the gateway has to know who is making the request. It could be an employee using an internal application, another company system or an AI agent working through an approved service. Once it knows who is asking, it has to check what they are allowed to do.
For example, a development team may be permitted to employ multiple authorised AI models whereas an application that processes sensitive information may be limited to one. Those permissions also should be separate from administrator access.
A user who is allowed to make queries to an AI model shouldn't be allowed to change the parameters of the gateway, the spending limit, or the provider credentials without further permission. The gateway also needs to connect securely to AI providers. Applications usually use API keys to prove they're authorised, but the gateway can manage these centrally instead.
Businesses still need to check how keys are stored, replaced and disabled, so applications can access the services they need without controlling the provider accounts.
Routing, policy checks and request limits
Once the gateway approves a request, it has to decide which AI model or provider should handle it. We refer to this as model routing. Routing depends on which providers are available, which models are approved and what the application needs.
A business might keep a second provider available when its usual one runs out of capacity, but it still can't send sensitive information to a provider that isn't authorised to process it. It also has to regulate how much AI any app can employ. That can mean limiting the number of queries an application can send, or the number of tokens it can consume, or the amount of money it is authorised to spend.
Tokens are the little units of text used by AI providers to assess usage and compute prices. The Microsoft documentation on AI gateway features includes how enterprises may use API Management to manage access, route requests to available models and track token usage. Spending limits are trickier because the gateway can't always predict how many tokens the AI's response will use.
It can block an application that's already reached its limit, but an approved request might still push spending over that limit. Businesses need to know how their gateway handles both situations.
What Must the Gateway Control During and After Inference?
The gateway's work doesn't end when it approves a request. During inference, when the AI model processes a request and generates a response, some controls need to work immediately. Others depend on information that only becomes available afterwards.
Latency, streaming and failure handling
Every check the gateway makes can add latency, or delay, to a request. This is especially noticeable when an AI model streams its answer in chunks. If the gateway checks every piece before passing it along, the application might feel slow. PayPal's October 2026 study shows one way to manage these delays. Its Cosmos LLM Gateway keeps slower security checks separate from the work that needs to happen immediately.
In tests using a simulated AI model, the gateway added a median delay of four milliseconds, rising to 11 milliseconds at the 99th percentile. These results are specific to PayPal's system, but they show why businesses need to measure the delays their own gateway adds. The gateway also needs a contingency strategy if an AI provider doesn’t respond, denies a request or is at capacity.
It may retry a failed request, but it shouldn’t retry indefinitely. A circuit breaker prevents that by turning off requests to a failing service for a period of time. The gateway can also switch to another AI service when the usual one fails. This is called failover. Switching to the same model version may be straightforward, but a different model could change an application's answers or behaviour.
Token accounting, observability and cost attribution
After processing, the gateway should show which application made the request, which model answered it, how long it took and how many tokens were used. This helps businesses track their AI usage and costs. Things can get difficult because suppliers don't always charge for tokens the same way.
Some say there is a difference between text provided to the model, text created in the answer and information they have previously processed and kept for reuse (called cached input). OpenTelemetry generative AI conventions provide a standard approach to log AI request and token usage information.
This allows enterprises to more easily compare activity across multiple providers and to discover which applications are responsible for the charges. It’s particularly important for FinOps teams that assist companies in understanding and managing their technology spend. The State of FinOps 2026 survey indicated that 98 per cent of respondents were managing AI spending, up from 63 per cent in 2025.
But tracking spending isn't the same as controlling it. The gateway can record what a request cost, but it can't undo the spending afterwards. Businesses must also reconcile the gateway’s information with their provider bills. Differences in price, caching and how tokens are counted mean the results don’t always match.
Where Can AI Gateway Controls Break Down?
Even with the right controls, a gateway can struggle when lots of requests arrive at once. Say an application sends ten requests simultaneously. Each passes its budget check because the gateway doesn't yet know what the others will cost. By the time the responses arrive, the application may have exceeded its limit.
Microsoft's LLM token-limit documentation warns that simultaneous requests can temporarily exceed token limits for this reason. It might also get problematic if a corporation has gateways in several locations. If each gateway tracks usage separately, an application could be within its limit in one region, yet exceed the total business limitation across all regions.
Companies need to know if the limits are shared across gateways or applied separately. Then there’s the question of what happens if something the gateway relies on stops working. Imagine that the system responsible for checking permissions is down. Should the gateway hold requests until it can check them again?
Or should it pass the requests, even if it can’t guarantee their legitimacy? The two approaches are known as fail-closed and fail-open. Fail-closed means denying requests when a mandatory check can’t be performed. Fail-open means letting them through. Neither is appropriate for every case.
If the gateway cannot verify that an application is authorised, allowing requests through may become a security issue. But if a mechanism to collect utilisation reports is suddenly unavailable, halting all AI requests may create unnecessary interruption. Companies also need to consider what happens if the gateway itself fails.
Even if the AI providers are doing a perfect job, applications might not be able to reach them. That’s why a gateway requires backup systems, a recovery plan and explicit guidelines for dealing with outages. These are not things that organisations should just start thinking about when something breaks.
How Should Enterprises Evaluate an AI Gateway?
When choosing an AI gateway, businesses need to look beyond its list of features. It might handle everyday requests perfectly well but struggle when traffic increases, a provider fails or several applications reach their limits at once.
What should buyers ask a gateway provider?
First, figure out what exactly the gateway can control, and what other systems it needs to work with. Some useful questions are:
- Access: Is there a way to control which apps, teams and individuals can utilise particular AI models? Who is allowed to modify those permissions?
- Credentials: Where are provider API keys stored? How do you replace or disable them as needed?
- Routing: Can I limit the providers, regions, and model versions available to an application? What happens when its usual provider isn't there?
- Budgets: Can the gateway block requests when an application has gone over its spending limit? What if there are numerous requests coming in at the same time?
- Monitoring: Can we monitor how much AI each application is consuming and cross-reference those numbers with what suppliers are charging us?
- Operations: Who can change the rules of the gateway? Can we test modifications before they impact existing running applications?
Businesses should also question where the gateway will be running and how it will interface with their existing systems for managing users, permissions and monitoring. Some gateways are managed services, meaning that portions of the infrastructure are handled by the provider.
This can free up internal personnel to focus on other activities, but organisations still need to evaluate what features are accessible, where the service can operate and what responsibilities are left with them.
The responses should clarify which controls are provided with the gateway, which require configuration and which the business will have to manage itself.
What should teams test before production?
A supplier can tell you what its gateway is supposed to accomplish, but enterprises still have to test what it will do with their own applications. A good starting point is to send a request to an approved model and get a successful answer. That doesn't tell you much about when things go south. Before taking a gateway to production, teams should make sure that it can:
- Prevent unauthorised users or apps from accessing a model.
- Enforce usage limitations and spending limits when many requests arrive at once.
- Handle services that don't respond or refuse requests without making repeated failed requests.
- Follow the right rules when systems responsible for permissions or other checks become unavailable.
- Process typical application workloads including streaming responses with no appreciable impact on performance.
- Maintain accurate consumption records that can be reconciled with provider bills for internal reporting purposes.
Tests need to reflect what the business will do with the gateway. This means employing the same AI providers, request kinds and quantities of traffic that you expect your applications to see. Microsoft’s multi-backend gateway architecture guidance also recommends verifying model versions while setting up failover.
Switching to another version may restore your AI access, but it may also modify the behaviour of a program. Teams should also trial modifications to gateway rules before rolling them out across the organisation. A new expenditure restriction or routing rule could affect numerous applications at the same time.
There has to be a way to introduce changes gradually and then reverse them if they cause problems. It's not about proving that the gateway will never fail. It's about looking at what occurs when it does, whether the business can cope with the fallout, and whether the teams have enough information to remedy the problem.
Final Thoughts: Control Is Only Useful When It Holds Up
An enterprise AI gateway makes it easier to manage applications using different AI providers, but businesses need to know what it can actually control. Those controls must work before, during and after requests, even when traffic increases or something fails. As more applications depend on the same gateway, problems with it can affect more of the business.
That's why testing how it behaves and knowing who's responsible for keeping it running are so important. EM360Tech continues to look at the infrastructure options for corporate AI adoption, helping technology teams understand what they need to think about as their AI systems evolve.
Comments ( 0 )