One week ago today, Microsoft opened the new Bing chat experience to the public waitlist, a chat box bolted onto Bing.com that drafts email replies, summarizes long documents, and, if the screenshots circulating this week are any indication, occasionally argues with the people testing it. That is a consumer story. The enterprise question behind it, and the one that has filled our week, is whether a regulated organization can get the same capability through Azure OpenAI Service, inside its own tenant, without a single prompt leaving its control. Three clients, a regional bank, a state agency, and the IT security group at a hospital system, called within four business days of each other asking exactly that.
The honest answer, as of this month, is yes, with real caveats. Azure OpenAI Service reached general availability in January, which sounds like "go," but "generally available" in this case still means an access application, a limited model list, and a set of controls that most application teams have never had to configure before. This note is our landing pattern for regulated clients: what GA actually unlocks, what compliance asks first, and the specific network, filtering, and identity decisions we make before a single production prompt gets sent.
We are deliberately not writing about prompt engineering or use-case selection here. Those matter, but they are the easy part. The hard part, the part that determines whether legal lets this out of the sandbox, is landing the service itself.
What Azure OpenAI Service's GA actually unlocks this month
GA removed the old closed-preview gate for the service itself, but it did not remove gating entirely. Getting an Azure subscription approved for Azure OpenAI Service still means submitting the access request form and waiting; in our recent experience, anywhere from a few days to a couple of weeks, longer if your justification is thin or your subscription is brand new. Regulated clients should plan the application into the project timeline, not treat it as a formality that happens the day after the kickoff meeting.
Once approved, the model list is short and worth being explicit about, because more than one client has assumed GPT-4 was already available because a vendor's slide deck implied it. It is not. As of this month the deployable models are the GPT-3.5-Turbo chat model and the text-davinci completion models, plus the smaller and embedding models in the same family. That is enough to build a genuinely useful internal assistant, a document summarizer, or a support-ticket triage tool. It is not enough to build whatever a vendor demoed to your CEO using the public ChatGPT interface, and we say that plainly in the kickoff so nobody is surprised in week three.
Regional availability is also narrower than most Azure services. We are deploying resources in a small number of US and EU regions right now, and clients with strict data-residency requirements need to confirm their preferred region is actually on the list before they promise a timeline to a steering committee.
The question compliance always asks first
Every regulated-industry kickoff starts the same way: where does the data go, and who can see it? For Azure OpenAI Service the answer starts with the tenant boundary. Prompts and completions run inside your Azure subscription, in the region you choose, and Microsoft's contractual terms for the service state that your prompts and completions are not used to train or improve the underlying models. That is the sentence that gets highlighted in every legal review, and it is the single biggest reason regulated clients choose the Azure service over calling a public API directly from an application.
It is not the whole story, and we make sure clients hear the rest of it before they sign off. By default, Microsoft retains prompts and completions for a period of time so that a small number of authorized employees can review flagged content for abuse, think generated content that looks like it's being used for fraud, harassment, or attempts to break the safety system. That review is narrow and audited, but for some clients, anything touching privileged legal content, protected health information, or classified-adjacent government workloads, "narrow and audited" is still not good enough. For those cases, there is a separate application process to request modified abuse monitoring, which removes Microsoft's human-review retention for an approved workload. We now build that second application into the project plan by default for any client with a regulator looking over its shoulder, because discovering the requirement in week six costs more than asking for it in week one.
Keeping the resource off the public internet
The Azure OpenAI resource is, by default, reachable over a public endpoint protected by key or Azure AD auth. That is fine for a proof of concept and wrong for anything a bank's network security team will approve. Our standard landing pattern disables public network access on the account and exposes it only through a private endpoint into the client's existing hub-and-spoke topology, the same pattern we already use for Key Vault and storage accounts.
New-AzCognitiveServicesAccount -ResourceGroupName "rg-aoai-prod" `
-Name "aoai-regulatedco-prod" -Type "OpenAI" -SkuName "S0" `
-Location "eastus" -PublicNetworkAccess "Disabled"
New-AzPrivateEndpoint -ResourceGroupName "rg-aoai-prod" `
-Name "pe-aoai-regulatedco" -Location "eastus" `
-Subnet $subnet -PrivateLinkServiceConnection $plsConnection
Two things trip people up the first time. First, private DNS: if the private endpoint's zone is not linked into every VNet that needs to resolve it, developers get a working name resolution from the jump box and a timeout from the actual application subnet, and lose a day to it. Second, anything calling the API from outside the network perimeter, a SaaS orchestration tool, a low-code connector, a developer's laptop over a coffee-shop hotspot, simply stops working once public access is off, which is the point, but it needs to be communicated before the switch is flipped, not after the on-call pages someone.
Content filtering is a control, not a preference
The service applies content filtering to every prompt and every completion, across four categories, hate, sexual, violence, and self-harm, each scored on a severity scale, with medium and high severity blocked by default in both directions. That is true whether you want it or not; there is no toggle on the resource blade to turn it off. For most regulated workloads that is a feature, not friction; it is a documented, vendor-provided control that shows up nicely in a control matrix next to your existing DLP and content-inspection tooling.
It does mean legitimate business content occasionally gets blocked: a claims-processing prompt describing an accident, a clinical note describing a self-harm risk assessment, a legal document quoting violent testimony. We tell every client to expect this in week one of the pilot, not as a bug to escalate but as a workflow question: what does the application do when the filter fires? Silently retry, return an unable-to-process message, or route to a human queue? Deciding that before the pilot starts saves an awkward incident review after a claims adjuster hits a wall in front of a customer.
Clients with a genuine business need for a different filtering threshold can apply for modified content filtering, the same way they can apply for modified abuse monitoring. Both applications ask for a description of the workload and its safeguards, and both are reviewed by Microsoft before approval; build the lead time in.
The governance wrapper we require before a pilot gets a green light
None of the above matters if the application still authenticates with a resource key pasted into a config file. Our baseline for every Azure OpenAI landing, regulated or not, is Azure AD authentication against the resource instead of key-based auth, with the calling identity, a managed identity where the workload runs on Azure compute, granted the Cognitive Services OpenAI User role rather than a broader Contributor assignment. Keys still exist as a fallback for tools that cannot do Azure AD auth yet, and where we cannot avoid them, they live in Key Vault with rotation on a schedule, never in application source or a pipeline variable typed in by hand.
We also insist on separate resources for non-production and production, not separate deployments inside one account. It costs a little more in quota administration, and it is worth every bit of the overhead the first time a load test against "dev" would otherwise have shown up in the same abuse-monitoring pipeline as real customer prompts.
| Control | Default in the portal | What we set for regulated clients |
|---|---|---|
| Network | Public endpoint, key or Azure AD auth | Public access disabled, private endpoint only |
| Auth | Resource key | Azure AD auth with managed identity, least-privilege role |
| Content filtering | On, fixed thresholds | Left on; exception request only with a documented business case |
| Abuse-monitoring logging | Microsoft retains for review | Modified abuse monitoring application for sensitive data classes |
| Environments | Single account, mixed use | Separate non-prod and prod resources |
That table is the actual checklist we walk a client's security team through before we let a pilot touch anything resembling production data. Every row has an owner by the end of the meeting, or the pilot date moves.
What we are telling clients to pilot first
Given the model list and the controls above, the workloads that make sense right now are narrower than the vendor pitch decks suggest: internal document summarization against content you already control, a support-ticket triage assistant that drafts a response for a human to edit, an internal knowledge-search assistant with retrieval grounded in your own SharePoint or file share content rather than the model's general knowledge. All three keep sensitive data inside the same trust boundary you already govern, and all three fail gracefully; a bad summary or a bad draft response gets caught by the human in the loop before it reaches a customer or a regulator.
What we are steering clients away from, for now, is anything customer-facing and unsupervised, and anything that assumes reasoning ability the current model list cannot deliver. That is not a permanent position; the model list will grow, and we expect it to, but landing this service well this quarter means being honest about what today's models and today's controls can actually carry.
If you're facing this
If your board saw the Bing chat demo this week and asked why you don't have "the AI thing" yet, the answer is not a rushed pilot against the public API from inside your production network. Land the service properly first: access, private networking, filtering, and identity, and the pilot conversation gets a lot shorter and a lot more defensible. If you are weighing an Azure OpenAI Service landing against a compliance review that has more questions than your team has answers, we are happy to compare notes on what we are building for other regulated clients right now.