API gateway pattern for AI applications

API gateways work fine for REST APIs. Connect it to an LLM and watch the costs spiral while your monitoring shows nothing useful.

That’s the problem. Most teams discover it after spending money they didn’t plan to spend.

Enterprise AI agent adoption is surging, but the industry forecast is sobering: over 40% of those projects face cancellation by 2027 due to runaway costs and complexity. Most of those teams will spend real money learning what traditional gateways can’t handle.

Why traditional gateways fail with AI

Traditional API gateways count requests. They rate limit by calls per minute, track response times, and flag error rates. That made sense when each request cost roughly the same and took similar time to process.

AI breaks all of these assumptions.

A single LLM request can consume 10 tokens or 10,000 tokens. The first request might cost a fraction of a cent. The second? Several dollars. Token consumption can vary by over 1000x between requests to the same endpoint.

Response times are just as unpredictable. A short answer takes 200 milliseconds. A detailed analysis runs 30 seconds or more. Traditional timeout settings either fail too fast or wait forever.

One user makes 10 requests, hits your rate limit, but consumed only 100 tokens total. Another makes 3 requests and burns through your entire daily budget with massive context windows. Your gateway can’t tell the difference, and it won’t try to.

The cost tracking problem

Running AI without token tracking is like paying for electricity without a meter. You find out the damage when the bill arrives.

Organizations try to manage LLM costs with traditional monitoring. It tells them nothing useful about who’s spending what, or why. TrueFoundry’s breakdown captures this problem well.

So what does proper tracking actually require? The AI gateway pattern needs to count tokens, not requests. It needs to track costs per user, per feature, per team. That means intercepting every request, parsing the prompt to count input tokens, reading the response to count output tokens, and multiplying by the current rate for that specific model.

Different models charge different rates. GPT-4o costs more than GPT-4o-mini. Claude Opus costs more than Claude Haiku. Your gateway needs to know which model handled each request and apply the right pricing.

Langfuse’s token tracking shows what production-ready cost management looks like. With 7M+ monthly SDK installs and status as the most-used open-source LLM observability tool, they’ve become a solid reference. They track tokens at the request level, aggregate by user and feature, and provide daily metrics for showback and chargeback. Without this, you can’t answer the basic question: which feature is burning through your AI budget?

Multi-model routing and fallbacks

Without fallbacks: you call OpenAI’s API, it returns a rate limit error, your application shows an error to the user.

With proper multi-model routing, the same scenario plays out differently. Your gateway automatically retries with Anthropic. User sees nothing. You stay online.

Portkey’s fallback patterns explain the implementation. Define your primary model, list fallback options in order, set retry logic and circuit breakers, and let the gateway handle failures automatically.

This gets more useful when you optimize for cost and performance at the same time. Route simple queries to faster, cheaper models. Send complex requests to more capable ones. If the expensive model is unavailable, fall back to the cheaper option instead of failing.

The Apache APISIX team documented how they handle multi-provider routing: proxying requests to OpenAI, Anthropic, Mistral, and self-hosted models through a single endpoint, with consistent authentication and rate limiting across all providers, and unified observability regardless of which model processed the request. Load balancing also helps when you’re hitting rate limits. Split traffic across multiple API keys for the same provider, distribute requests across models with similar capabilities, and route to different regions based on latency.

Security and audit requirements

API key management for AI gets complicated fast. Each developer needs keys for testing. Each environment needs different keys. Each customer might need isolated keys for compliance.

I think most teams underestimate this part until something actually goes wrong. Storing keys in application code is obviously wrong. Environment variables are barely better. API gateway security patterns show that proper key management means storing credentials in a secure vault, rotating them regularly, using the gateway to inject keys at request time, and never exposing raw keys to client applications.

Data privacy matters more with AI than with traditional APIs. Every prompt you send potentially contains sensitive information. Every response might include data you shouldn’t cache or log. The gateway needs to sanitize logs, remove personally identifiable information before storage, enforce data residency rules, and support compliance requirements like GDPR and HIPAA without making developers implement these controls in every application.

Audit logging becomes critical here. Who made which request? What data did they send? Which model processed it? How long was the response cached? These questions come up during security reviews and compliance audits. Your gateway should answer them without you digging through application logs.

What the production pattern looks like

Real implementations from Apache APISIX users show a consistent approach. Companies like Zoom, Lenovo, and Amber Group use API gateways to manage AI traffic alongside traditional APIs, but configure them differently for AI workloads.

Kong Gateway offers token-based rate limiting that counts tokens instead of requests. Their implementation pulls token data directly from LLM provider responses, supports limits by hour, day, week, or month, and handles different limits for different models.

Azure API Management shows enterprise-grade AI gateway in practice. They manage multiple AI backends from a single gateway, implement semantic caching to reduce duplicate requests, provide built-in token metrics and cost tracking, and integrate with existing API management workflows.

Semantic caching is worth prioritizing. Semantic caching systems demonstrate 61-69% cache hit rates in production deployments, which makes caching a real cost opportunity. Multi-tier caching architectures combining semantic caching with provider-level prompt caching can reduce costs by 80% or more. Anthropic’s prompt caching alone offers up to 90% cost reduction for long prompts. OpenAI’s automatic caching delivers 50% savings by default.

The self-hosted versus managed decision depends on your priorities. Self-hosted gives you complete control over data routing and security policies. Managed solutions cut operational overhead but lock you into a single vendor’s platform.

Observability comes first. If you can’t see token usage, model performance, and cost attribution, you can’t optimize anything. 89% of teams running AI agents have implemented observability, outpacing evaluation adoption at 52%. What good observability for AI gateways covers: token consumption per request, cost per user and feature, cache hit rates, model latency and error rates, and how often fallbacks actually trigger.

Skip the gateway and you’ll spend six months recovering from a bill that could have been a dashboard alert.