Building an AI Landing Zone on Azure - Part 1: Why every enterprise needs an AI gateway

This is part 1 of a five-part series on building an AI Landing Zone on Azure. Part 1: Why every enterprise needs an AI gateway (this post) · Part 2: The platform underneath · Part 3: One front door for many models · Part 4: Metering every call · Part 5: Turning tokens into euros

Over the last year I have been building and running an AI Landing Zone for a regulated financial services client operating under DORA. It took considerably more effort than I expected when I started, and most of that effort went into things that are not in any Microsoft quickstart. So I wanted to write it down, because I think the pattern is going to be needed by a lot more organisations than currently realise it.

This first post is about the why and the overall solution. The next four go deeper into the platform, the gateway, the metering pipeline and the chargeback reporting.

The problem: AI adoption is happening whether you plan for it or not

If you work in a mid-size or large organisation, you have heard the phrase "we need to do something with AI" more than you want to. Somebody knows a great company they have worked with before, that can implement a PoC in a couple weeks with pre-configured resources (in IaC if you are "lucky") expecting to integrate with your business systems with a click of a button. Nobody seems to care about security, compliance or governance; speed is the message.

Three months later there are a number of these PoCs running. Nobody knows which ones are in production. Nobody knows what they cost, because the bill is one line item per resource and the resource is shared by three teams. Nobody knows what data is being sent to which model. And when the auditor, or in our case the DORA-driven ICT risk function, asks "which third-party AI services do you depend on, how are they authenticated and what happens if one goes down?" the honest answer is "we don't know".

We all hear this story quite often. It is not a technology failure, it is a platform failure: there was no solution available, so people built their own.

What "landing zone" means here

The general term landing zone means a pre-configured, governed environment that workloads can land in. An AI Landing Zone applies that idea specifically to the consumption of AI models. The goal I set was simple to state and hard to deliver:

Every call to any AI model goes through one private front door, is authenticated with an identity rather than a key, is inspected for safety, is metered, and is attributed to the team that made it, in euros.

Everything else in the architecture follows from that sentence.

The architecture in one picture

Read it left to right. End users reach the frontend applications running on the Kubernetes cluster through an Application Gateway with a WAF in front. Those frontends talk to backend workloads in the same cluster (protected by Cilium network policies) and it is the backends, along with developer tooling on the private network, that call the AI gateway: Azure API Management in its most locked-down configuration, Premium v2, injected into the VNet, no public IP. Consumers authenticate to APIM with an API subscription key that identifies their product and team. APIM validates it, strips it, checks the request against Content Safety, acquires a token with its own managed identity and forwards the call to the right model backend over a private endpoint.

On the way back, APIM emits a usage event: which subscription, which product, which model, how many prompt tokens, cached tokens, completion tokens, reasoning tokens, or for the OCR model, how many pages. That event lands in Event Hub, gets picked up by a small Function App, and is written to Cosmos DB. A rate card, also in Cosmos, prices it. Power BI turns it into a report that the team actually uses.

The whole thing is Terraform. One codebase, two environments, deployed by a GitLab pipeline authenticating with OIDC. No click-ops.

The design principles, and why each one is there

I want to spell out the principles because they explain almost every decision in the later posts. Each one exists because of a real requirement, not because it looked good on a slide.

Private endpoints everywhere. Every PaaS service in the zone, from Cosmos DB to Event Hub to the model endpoints, has public network access disabled and is reachable only through a private endpoint in the VNet. Egress leaves through the hub firewall. For a financial company this is not optional, and it also turns out to be the source of the most interesting challenges, which I will get to in part 4.

No shared keys, anywhere. Storage accounts, Cosmos, Event Hub, Key Vault: shared key authentication is disabled. Every component authenticates with a user-assigned managed identity, and every identity has exactly the roles it needs. The APIM identity can send to Event Hub but not read from it. The Function App identity can read from Event Hub and write to Cosmos, but cannot call a model. When a security reviewer asks "what can this thing do if compromised", the answer fits on one line.

Caller credentials never reach the model. The consumer authenticates to APIM with an Entra ID token or an APIM subscription key. APIM throws that away and authenticates to the backend with its own identity. The model endpoint has no idea the consumer exists. That means model keys do not exist in any application's configuration, and rotating or replacing a backend is invisible to consumers.

Metering is a platform concern, not an application concern. Applications do not report their own usage. The gateway observes it. That is the only way to get numbers you can trust for chargeback, because nobody can forget to instrument, and nobody can under-report.

Cost is computed at query time, from data. The pipeline stores raw usage, never prices. The rate card is a table. When Microsoft changes a price, or a new model appears with a separate rate for cache writes (which happened during the build), you update one row and history re-prices itself. I have seen enough "we hard-coded the price in the ingestion function" pipelines to know how that ends.

One codebase, many environments. Non-production and production are the same Terraform with a different variable. Anything that differs per environment, like the Event Hub endpoint the APIM policy posts to, is a named value or a local, never a copy of the code.

What is actually in the AI Landing Zone

To make this concrete, here is the inventory. I will go through each of these in the coming parts, so this is just a quick summary.

The network is a spoke VNet in a hub-and-spoke topology over Azure Virtual WAN, with dedicated subnets for the APIM injection, the Function App integration, private endpoints, the application gateway, the Kubernetes nodes, and the database. Every subnet has an NSG with an explicit deny-all at the bottom.

The platform is a private AKS cluster running Azure Linux, Cilium, workload identity and Karpenter node auto-provisioning, with ArgoCD doing GitOps for everything that runs on it. Two internal applications live there today, each with their own identities, Key Vault and PostgreSQL database.

The gateway is APIM Premium v2, VNet-injected, with a library of reusable policy fragments and a product per consuming team. It fronts Azure OpenAI (the gpt-5 family, both Chat Completions and the Responses API) and Mistral Document AI for OCR through an AI Foundry serverless endpoint. Supporting services the applications need, such as Azure Maps for geocoding, live in the zone under the same rules: private endpoint, local authentication disabled, managed identity only.

The metering pipeline is APIM → Event Hub → Flex Consumption Function App (Python) → Cosmos DB, with a side channel into VictoriaMetrics for live dashboards.

The reporting is a Cosmos rate card in EUR plus a Power BI / Fabric report that computes cost per product, per subscription, per model, per day.

And the governance wrapper is Terraform with pre-commit hooks running tflint, checkov and secret detection, a GitLab pipeline with OIDC federation to Azure, and remote state shared with the rest of the client's landing zones.

Was it worth it?

Yes, and the reason is not the technology. The reason is what changed organisationally once it existed.

Teams stopped asking for their own OpenAI resource, because getting an APIM subscription in the AI product is faster and comes with a dashboard. The finance team got a daily view of AI spend by user for the first time. The risk function got a single answer to "where do we call AI models from and how", with a diagram they could put in the DORA register. And when a model version turned out to be unstable in preview, I swapped the backend and nobody outside the platform team noticed.

None of that is possible when every team has its own key.

What comes next

In part 2 I will cover the platform underneath: the network, the private AKS cluster, GitOps with ArgoCD, and how workload identity replaces every secret an application would normally need. Part 3 is the gateway itself. Part 4 is the metering pipeline. Part 5 is the chargeback model and the report.

If you are looking at a similar problem, whether it is the regulatory angle, the cost angle or just the "we have eleven OpenAI resources and no idea what they do" angle, I am packaging this landing zone as an accelerator that can be deployed into an existing Azure estate in a matter of weeks rather than months. Feel free to connect with me on LinkedIn or reach out via ascode.nl if you would like to talk it through.

Thanks for reading, see you in part 2.