We have run enough generative features from proof of concept to production over the past year to have turned the exercise into a checklist, and the checklist has earned its keep. A demo persuades a steering committee; it does not survive contact with real users, adversarial inputs, and an on-call rotation. The gap between those two states is where security reviews live, and for AI features that gap has a specific shape that a traditional application review misses. This is the pass we now run before any AI feature is allowed to carry a general-availability label, in the order we run it, because the order matters as much as the contents.
The framing we bring to it: an AI feature is a system that takes untrusted input, makes a probabilistic decision, and may take action on the result. Every item below falls out of that sentence. Traditional app security assumed deterministic behavior and trusted-once-authenticated flows; generative systems break both assumptions, and a review that does not account for the breakage will pass a feature that is genuinely unsafe. None of this is about slowing teams down. A feature that ships and then causes an incident is far slower than one that clears a real gate first.
Start with the data-flow diagram, or start over
Nothing else on this list means anything until you can draw where the data goes, and we will not begin a review without one. The diagram we insist on shows every hop: user input in, through the front end, into whatever orchestration builds the prompt, out to the model, back with a completion, and then — the part teams always underdraw — everywhere that completion travels afterward. Which data stores feed the grounding. Which identity reads them. Where the output is rendered, logged, or passed to another system. What leaves your trust boundary and what stays inside it.
The diagram is not busywork; it is where the review actually happens. Drawing it surfaces the questions that matter: this retrieval step reaches a store holding regulated data, so who can see it. This output flows into a system that can take action, so what happens when the output is adversarial. This log captures the full prompt, so is it now a store of sensitive input nobody classified. We have caught more real problems in the hour spent drawing the diagram than in any other single part of the review, because a system nobody can draw is a system nobody fully understands, and you cannot secure what you do not understand.
Prompt injection: defense in depth, because no single layer holds
Prompt injection is the signature vulnerability of this class of system and there is no clean fix, which means the only responsible posture is layers that each assume the previous one failed. We check for all of them, because a review that finds one control and stops has found a false sense of security.
- Input handling. Untrusted input — direct from the user, and critically the content retrieved into the prompt as grounding — is treated as untrusted. Retrieved content is fenced and labeled as reference data rather than instructions, and inputs are size-capped so a single payload cannot dominate the context or the bill.
- Output encoding. Model output is untrusted user input, full stop, because a successful injection makes it exactly that. It is encoded before rendering so it cannot carry a script into the page, and it never reaches an interpreter, query, or shell without validation in between.
- Tool-call allowlists. If the model can invoke tools, the tools are an explicit allowlist, each with validated parameters and the narrowest capability that does the job. The model does not get a general-purpose action surface; it gets a short menu.
- Least-privilege grounding. Retrieval runs with, at most, the asker's permissions, never a broad service identity that can read everything. The security-trim happens at retrieval, so the model can only ever summarize what the user was already entitled to see.
The mental model we hold the team to: assume the injection succeeds, and ask what it can accomplish. If the answer is "read data the user could already read, and render text that gets encoded," you have contained it. If the answer is "call a tool that moves money," you have not, and no amount of clever prompt-side filtering changes that.
Identity, secrets, and the platform floor
This part is not novel and it is where we still find the most findings, because teams treat AI as exotic and forget the fundamentals underneath it. The AI is running on a platform, and the platform has an identity story that has to be right first.
Managed identities for service-to-service authentication, so there are no keys to leak in the first place. No secrets in code, ever — not in the repository, not in the container image, not in an environment variable checked into config. Anything that must be a secret lives in Key Vault and is pulled at runtime by an identity with access to exactly that secret and nothing else. The endpoints fronting the model are on private networking so they are not reachable from the public internet. This is the same floor we demand of any production workload; the model does not earn an exception because it is interesting. We have reviewed genuinely sophisticated AI applications that fell at this hurdle — a real API key in a config file, a service principal with subscription-wide rights doing a job that needed read access to one container — and a clever architecture does not survive a leaked credential.
Monitoring, logging, and the eval gate
You cannot operate what you cannot see, and generative systems are harder to see into than most because their behavior is probabilistic. We require abuse monitoring and logging built for that reality: enough captured to reconstruct what happened when something goes wrong, without the log itself becoming an unclassified store of the sensitive inputs users typed. That balance is a real design decision, and getting it wrong in either direction — logging too little to investigate, or logging too much and creating a new liability — is a finding. Rate limits and anomaly alerting catch the abuse and the runaway-cost cases before they become the story, and cost is a security property here because token-metered systems can be attacked through the invoice.
Then the gate that most teams skip and that separates reliable systems from lucky ones: evaluations. Before a feature ships it runs against a golden dataset of representative inputs with known-good outputs, scored for the properties that matter — groundedness, relevance, refusal to leak. Alongside the goldens sits a jailbreak-and-injection suite: the adversarial prompts designed to make the system misbehave. A regression on either set blocks the release the same way a failing test blocks a merge. Without this gate every change is a bet, and the day a prompt tweak that fixed one case quietly broke five, you find out from a user instead of from CI. Build the eval before the feature; it is the difference between engineering and hoping.
The red-team pass, the kill switch, and the human in the loop
Three things close the review, and a feature does not get its GA label without all three.
First, a red-team pass by someone who did not build it. The builders know the happy path too well to attack it convincingly; you want a person trying to make the system leak, act badly, or embarrass you, working from the OWASP LLM categories as a map. Findings feed back into the eval suite so today's manual discovery becomes tomorrow's automated gate, and the red team gets meaner every release.
Second, a rollback and kill switch. Every AI feature ships with a documented, tested way to turn it off fast — a flag, not a redeploy — because generative systems fail in ways you did not anticipate and the ability to disable a misbehaving feature in seconds is worth more than any pre-launch guarantee. We test the kill switch before launch, because an untested kill switch is a hope, and the moment you need it is the worst moment to discover it does not work.
Third, human-in-the-loop for consequential actions. The single most important design decision in the whole system is how much agency the model has, and the rule is simple: the higher the stakes of an action, the more a human sits between the model's decision and the real-world effect. Reading and drafting can be autonomous. Sending, filing, paying, deleting, provisioning — anything with a consequence that is hard to reverse — the model proposes and a person disposes. This is not a lack of ambition; it is the recognition that a probabilistic system will eventually be confidently wrong, and you want that wrongness to hit a human review step rather than production.
A 7,000-seat professional-services firm ran a client-facing summarization feature through exactly this pass this spring, and the review earned its fee twice over. The data-flow diagram exposed a log capturing full prompts — including client material — into a store nobody had classified. The red-team pass got the summarizer to follow an instruction planted in a retrieved document. Neither was catastrophic, both were quiet, and both would have shipped without the pass. They fixed the log, tightened output handling, added the injected instruction to the eval suite, and went to GA a week later than planned and considerably safer, which is the trade every serious team should be willing to make.
The compliance clock is now part of the plan
One paragraph on the regulatory backdrop, because it is now real enough to plan against rather than watch. The EU AI Act was approved by Parliament in March and is expected to enter into force this summer, with obligations phasing in over the following months and years by risk tier. For a practitioner the takeaway is not to panic and not to ignore it: know which of your AI features would land in which risk category, keep the documentation the higher tiers will require — data provenance, evaluation results, human-oversight design — as a byproduct of building well rather than a scramble later, and treat "we can explain and evidence how this system behaves" as a durable requirement regardless of which jurisdiction's rules bite first. Most of what the Act's higher tiers demand is what a good review already produces; the work is keeping the paper, not inventing new controls.
The context around all this keeps accelerating — GPT-4o landed last month and reset expectations for what these systems feel like, and the first Copilot+ PCs are literally shipping to customers today. The models will keep getting better and the surface will keep getting wider. The review does not, which is the point: capability changes fast, and the discipline that makes capability safe to ship changes slowly, so a checklist that is boring on purpose is exactly what you want between a demo and your users.
If you're facing this
If you have an AI feature approaching a launch date and the security review so far has been someone trying a few prompts and pronouncing it fine, the gap between that and a pass your on-call team can stand behind is the whole list above. We run this review as a fixed engagement and would rather find the log capturing client data before it ships than after. Reach out and we will run the pass with your team before the GA label goes on.