Compliance paperwork loves neat diagrams. LLM features hate them. The gap shows up in security reviews when someone asks, “What happens if the retrieved document tells the model to ignore policy?” and the answer is a paragraph about “responsible AI principles.”
This note is the threat-modeling pass we run on generative AI and agent features — usually before SOC 2 evidence collection, not after the auditor’s question lands in Slack.
Prompt injection is a systems problem
Treat injection like XSS: assume untrusted text will try to become instructions. Untrusted text includes:
- User messages.
- Retrieved documents and tickets.
- Web pages and email bodies your tools fetch.
- Other users’ content in multi-tenant corpora.
Mitigations stack; none alone is enough:
- Architectural: do not give the model tools that can exfiltrate or administer based solely on natural language from untrusted channels.
- Retrieval: separate system prompts from content; mark untrusted content explicitly in the prompt structure; prefer structured tool outputs over pasting raw HTML.
- Output: allowlists for outbound links and actions; blocklist theatre alone will fail.
- Monitoring: detect anomalous tool sequences and sudden spikes in external calls.
If your agent can send email or call an admin API, injection is not theoretical. It is the primary abuse case.
Data flow boundaries you must draw in ink
Before model choice, draw:
- What leaves the tenancy boundary (to a model provider, to logs, to eval stores).
- What is retained (prompts, completions, embeddings) and for how long.
- Who can access traces (support, eng, vendors).
- Whether training / abuse monitoring on the provider side is off for your tier.
On Azure OpenAI and similar enterprise offerings, get the contractual and configuration facts straight — region, logging, abuse monitoring options — and put them in the architecture decision record. Security reviewers will ask. Product should not discover the answers during the review.
OWASP LLM Top 10 as a review agenda, not a poster
We walk the OWASP LLM risks as a meeting agenda with owners:
| Risk theme | Questions we force |
|---|---|
| Prompt injection | Where does untrusted text enter? What can the model do after? |
| Insecure output handling | Do we execute or render model output unsafely? |
| Supply chain | Which models, SDKs, and plugins are pinned and reviewed? |
| Data disclosure | Can retrieval cross tenants? Can logs leak PII? |
| Excessive agency | Which tools are irreversible? Where is HITL? |
| Overreliance | Where does the UX overclaim certainty? |
| Model DoS / cost | Rate limits, quotas, max steps? |
| Sensitive info in prompts | Are secrets ever concatenated into context? |
You do not need a 40-page report. You need a risk register with severity, mitigation, residual risk, and a named owner. That document is more valuable than a generic “AI policy” PDF.
Identity still matters more than the model brand
Most serious failures we see are identity and tenancy mistakes wearing an AI costume:
- Shared API keys in a server that does not enforce user context on retrieval.
- Service principals with broad Graph or data-plane rights “for the demo.”
- Eval datasets copied to laptops with production tickets intact.
Entra ID, managed identities, Key Vault, private endpoints — the boring Azure controls — are still the backbone. The LLM does not excuse skipping them.
What “SOC 2 ready” means for an AI feature
Auditors vary, but engineering evidence that repeatedly helps:
- Change control for prompts, tools, and model versions (git + approval).
- Access control evidence for admin UIs and trace stores.
- Data retention and deletion paths for prompts/completions if you store them.
- Incident runbook: disable flag, rotate keys, review traces.
- Vendor reviews for model providers and observability SaaS.
“We use a safe model” is not a control. “Here is how we prevent cross-tenant retrieval and who can turn the feature off” is a control.
If you are facing this
Send a one-page data flow (even a rough one) and the tool list for the feature. We will return a short risk register and a remediation order — fixed-scope security review, not a multi-month transformation program unless you actually need one.
Extended practice notes (2026)
The remaining gap between a short checklist and a usable field note is usually scenario detail. In practice, the same engagement type described above still requires explicit answers to: what is in scope this quarter, what is deferred with a date, who can halt a wave, and how success is measured in production — not in a lab.
We document those answers before the first production change. When stakeholders disagree, the disagreement is resolved in writing, not on the bridge at cutover. That habit is independent of whether the workload is mail, identity, virtualization, desktop delivery, or security hardening.
Repeatable detail also includes communication: who tells users what changes, when the freeze starts, where status is posted, and how exceptions are requested. Technical excellence without communication still produces an outage from the user's point of view.
If you're facing this
If you are facing this class of problem, bring inventory and a sponsor who can decide. We still take this work.