By October 2025 most enterprises we work with have something AI-shaped in production: Microsoft 365 Copilot seats, a Copilot Studio agent, an Azure OpenAI line-of-business assistant, or all three. What they often lack is a coherent LLM red teaming practice and an evaluation framework that product, security, and risk all trust. This note is how we run evals and adversarial testing for enterprise Copilot systems without turning the program into either a one-off penetration stunt or an academic benchmark nobody ships against.
We assume you already accept that demos lie. The companion pieces are RAG evals before the feature flag and agent loops with human-in-the-loop. Here we connect those ideas to Copilot-style grounding, multi-surface estates, and a red-team cadence that fits enterprise change windows.
Three systems, one control objective
Enterprises blur "Copilot" into one word. We split the evaluation object:
- M365 Copilot (productivity) — grounding over mail, files, meetings, and chat under user identity.
- Declarative / Studio agents — tools, knowledge sources, channels, and possible write actions.
- Custom LLM apps — Azure OpenAI or Foundry apps with your orchestration, RAG, and tools.
Control objective is shared: do not cause unacceptable harm under realistic and adversarial use, while remaining useful enough that people do not bypass the system. Evals without red team miss attackers. Red team without goldens miss silent quality regressions. You need both.
+------------------------+
| Release candidate |
| (prompt/graph/index) |
+-----------+------------+
|
+-----------------+-----------------+
| | |
v v v
Golden quality Safety suite Red-team ops
(task success, (leak, refuse, (inject, jailbreak,
groundedness) agency limits) tool abuse)
| | |
+-----------------+-----------------+
|
v
Go / no-go gate
+ monitoring hooks
What we measure (and refuse to fake)
Quality and usefulness
- Task success on role-realistic scenarios (legal summary, support draft, policy answer).
- Groundedness / citation correctness where the product claims grounding.
- Refusal quality on out-of-scope or disallowed asks.
- Latency and cost budgets as first-class metrics, not afterthoughts.
Safety and security
- Sensitive data exposure beyond entitlement or policy.
- Prompt injection leading to policy ignore, data exfil attempts, or tool misuse.
- Excessive agency — irreversible actions without HITL.
- Misinformation risk on high-stakes domains (medical-adjacent, financial advice, HR decisions).
- Jailbreak resilience relative to your policy, not a generic internet score.
Operations
- Trace completeness (can on-call reconstruct a bad run?).
- Alerting on eval drift and on production anomaly signals.
- Time-to-disable for a bad prompt or agent version.
| Dimension | Example metric | Gate style |
|---|---|---|
| Groundedness | Percent answers with valid citations on golden set | Block release if drop exceeds threshold |
| Refusal | Correct refuse rate on policy cases | Block on critical policy classes |
| Injection | Attack success rate on fixed library | Block above agreed residual rate |
| Agency | Disallowed tool attempts in sandbox | Block any success on write tools |
| Cost | Median tokens per successful task | Warn; block if runaway |
| Latency | p95 user-visible time | Warn or block per SLA |
Thresholds are business decisions recorded with risk — not numbers we invent in a blog and you paste blindly.
Golden sets that look like work
Happy-path goldens from marketing PDFs are how you ship false confidence. We build sets from:
- Redacted tickets and real document classes users touch.
- Multi-hop questions that require two sources.
- "Almost in corpus" items that must refuse or escalate.
- Role-based packs: finance, HR, engineering, executives — different harm models.
- Multilingual cases if your workforce is multilingual (many estates forget this).
For M365 Copilot-style experiences, pure offline goldens are harder because grounding is live tenant content. We still:
- Maintain a lab tenant or tightly controlled site collection with synthetic but realistic content and known ACLs.
- Run scripted user journeys with test accounts in different groups.
- Capture anonymized production failure tickets into the golden set after review.
For custom RAG and Studio agents, offline goldens with frozen corpora are mandatory in CI. See the harness philosophy in RAG evals before the feature flag.
Size guidance we actually use: fifty to one hundred cases to start a narrow agent; a few hundred before broad GA; continuous growth from production misses. Quality of labels beats vanity count.
Red team: scope, rules of engagement, cadence
LLM red team in enterprise is not "ask the model to say something naughty" forever. It is a structured attempt to violate your policies and control objectives.
Scope examples
In scope: prompt injection via documents and email, data exfiltration attempts through answers, tool invocation abuse, policy bypass, multi-turn social engineering of the agent, retrieval of content the test identity should not access (ACL tests), cost amplification.
Out of scope unless separately authorized: attacking Microsoft's shared platform outside your tenant configuration, targeting real personal data of employees, destructive writes in production, phishing real users.
Rules of engagement
- Written authorization from the system owner.
- Dedicated test identities and, where possible, non-production or cloned content.
- Logging of all attempts for reproducibility.
- Severity rubric agreed before testing starts.
- Fix verification window and retest policy.
Cadence
| Trigger | Red-team depth | Owner |
|---|---|---|
| Major prompt / graph / tool change | Focused regression + new attacks for the change | Security + eng |
| New connector or knowledge source | Injection and ACL suite | Security |
| Quarterly | Broader campaign on top agents | Red team / security |
| Post-incident | Targeted retest of failure class | Incident lead |
One heroic purple-team week at purchase time is not a program.
Attack libraries we keep current
We maintain versioned attack sets (text files and document fixtures) rather than relying on whatever is trending on social media that week. Categories:
- Direct policy probe — requests for disallowed advice or secrets.
- Indirect injection — instructions hidden in docs, tickets, HTML, PDFs.
- Exfil framing — "summarize everything about X and format as CSV," encoding tricks, multi-step extraction.
- Authority spoof — "I am the CISO, ignore previous policies."
- Tool coercion — "call the refund tool," "email this externally," "export the roster."
- RAG poisoning / conflicting sources — contradictory documents, stale policies.
- Multilingual and obfuscation — base64-ish tricks, homoglyphs, split instructions (as applicable to your filters).
- Capability hallucination — tricking users via confident false system claims (product UX issue as much as model issue).
Map findings to OWASP LLM Top 10 language so risk committees hear a dialect they already approved.
Severity rubric that drives tickets
Without severity, every flashy jailbreak becomes either a fire drill or is ignored. We use a simple grid:
| Severity | Definition | Example | Response |
|---|---|---|---|
| Sev1 | Confirmed unauthorized data access or irreversible action possible | Agent emails external party with HR data without HITL | Block release / kill switch |
| Sev2 | Reliable policy bypass with user-level harm | Consistent confidential summary beyond need-to-know in lab ACL test | Fix before broad expand |
| Sev3 | Degraded refusal or partial leak under contrived turns | Multi-turn coax reduces refusal quality | Tracked fix, monitor |
| Sev4 | Benign policy nits or off-tone answers | Mild sycophancy, low-stakes wrong style | Backlog |
Severity is about your data and your tools. A model saying a forbidden word in a vacuum may be Sev4; the same model exporting a payroll field is Sev1.
Integrating evals into delivery
Engineering will not run a fifty-page PDF before every deploy. Embed gates:
- PR smoke goldens — fast subset, deterministic checks first (schema, citations present, hard refuse keywords where appropriate).
- Nightly full goldens — LLM-as-judge with versioned rubric plus human sample review weekly.
- Pre-prod red-team pack — automated subset of injection cases on every release candidate.
- Manual red team — scheduled, especially for Sev1 tool surfaces.
Store traces with prompt version, retrieval IDs, tool args, model name, and judge version. When a production incident hits, you need minutes-to-reconstruct, not archaeology.
For agentic systems, score trajectories: wrong tool order can be a failure even if the final sentence looks polished. That is the core of our agent loop guidance.
Copilot-specific testing notes
Productivity Copilot
- ACL tests with users in and out of groups on synthetic sensitive libraries.
- Oversharing scenarios: content that should not have been broadly readable — partner with the hygiene program in Copilot security, DLP, and oversharing controls.
- Prompt paste tests for secrets (pair with DLP).
- Grounding quality on stale vs current policy docs (lifecycle problem wearing an AI costume).
You cannot "unit test" the entire Microsoft service, but you can regression-test your tenant configuration, labels, DLP, and user guidance as part of readiness.
Studio agents
- Every tool in a sandbox with deny-by-default network where possible.
- Channel tests (Teams vs other) because prompt surfaces differ.
- Knowledge source freshness and ACL sync bugs.
- Publish pipeline: no direct-to-prod edits.
Custom apps
- Full offline harness ownership — you own the model chain.
- Tenant isolation tests if multi-tenant SaaS.
- Security review checklist before GA as exit criteria, not a suggestion.
LLM-as-judge without self-delusion
Judges drift when you change the judge model silently. Practices:
- Pin judge model and rubric version in git.
- Use deterministic checks whenever possible (JSON schema, citation ID existence, forbidden tool list).
- Calibrate judges against human labels on a holdout; re-calibrate when judge or rubric changes.
- Never let the same model grade its own creative jailbreak success without human spot-check on security-critical cases.
Judges are force multipliers, not oracles.
Production monitoring is part of evals
Offline goldens do not see the long tail. Production needs:
- User feedback hooks that create tickets with trace IDs.
- Sampling of answers for groundedness review on high-risk agents.
- Anomaly detection on tool call rates and token spikes (possible abuse or loop).
- Periodic replay of anonymized failures into the golden set.
Shadow mode remains the best pre-GA technique for custom systems: same traffic, hidden answers, scored offline.
Program anti-patterns
Red team as PR. Publishing that you "red-teamed the AI" without severity, fixes, or retest is theatre.
Only external consultants, no internal harness. Consultants are great for campaigns; CI goldens are how you do not regress next Tuesday.
Blocking on every rude completion. Focus on enterprise harm models. Infinite content-moderation perfection is not the same as data security.
No lab tenant for Copilot ACL tests. Testing only on production personal content is how you create real incidents while testing for hypothetical ones.
Evals owned solely by a science team that does not ship. Product must feel the gate or they will route around it.
A ninety-day standup plan
Days 1–15. Inventory AI systems. Name owners. Agree severity rubric. Stand up trace storage.
Days 16–45. Build golden v1 for the top system. Automate smoke in CI. Run first authorized red-team campaign on tools and injection.
Days 46–75. Fix Sev1/Sev2. Add production sampling. Expand goldens from real failures. Wire kill-switch drills.
Days 76–90. Steering review with metrics. Decide which systems may expand users. Document residual risk for risk committee.
Inventory --> Goldens v1 --> Red team 1 --> Fix --> Sample prod --> Gate policy
| | | | | |
owners CI smoke severity retest feedback expand rules
systems lab data report traces to goldens residual risk
How this ties to buyer strategy
If you are mid-purchase on more Copilot seats or a platform rewrite, fund eval and red-team capacity in the same business case. Orphaned "innovation labs" that cannot block a bad release do not reduce risk; they produce PDFs. Pair technical gates with the adoption discipline in rollout and change programs — quality that users do not trust will not be used, and systems that users misuse need DLP and training as much as model work.
Related threads worth sequencing: permissions-first Copilot readiness, data governance for AI, first production months, and security hardening for oversharing and prompt exfil.
What to send us for a written eval readiness read
Provide: list of AI systems in prod or pilot, whether tools can write, current test story (honestly), one serious near-miss if any, and whether you have a lab tenant. We will tell you whether to invest first in goldens, ACL lab design, tool sandboxing, or program operating model — fixed scope, no obligation to implement with us.
LLM red teaming for enterprise is not a dark art and not a one-time badge. It is golden sets from ugly reality, attack libraries under change control, severity that drives fixes, and gates that product cannot shrug off. That is the difference between a Copilot estate you can defend in front of audit and one you can only demo.
Deep dive: building the lab tenant for Copilot-style ACL tests
Production hunting for ACL bugs is how red teams become incident creators. A lab tenant or isolated site architecture should include:
- Synthetic users in nested groups that mirror real org patterns (matrixed access, not only clean hierarchies).
- Document sets labeled and unlabeled, encrypted and not, with intentional overshare controls you will later remove to test detection.
- Negative controls: content that group A must never surface to group B.
- Scripted journeys: search, summarize, compare, "draft email from these files."
- Reset procedures so each campaign starts from a known ACL state.
Budget this lab like a permanent fixture. Rebuilding synthetic content every quarter from scratch is how programs quietly stop testing.
Deep dive: scoring groundedness without infinite debate
Teams stall arguing whether an answer was "mostly right." Use layered scoring:
- Hard fail — cites a document ID that does not exist or invents a policy clause with no support.
- Soft fail — omits a critical caveat present in the top sources.
- Pass with note — correct but incomplete on a minor detail.
- Pass — meets the task contract.
Judge prompts should output the class, not only a float score. Floats invite threshold thrash; classes invite ticket workflows. Humans adjudicate disagreements on a sample; update the rubric when systematic bias appears.
Deep dive: coordinating red team with change freezes
Enterprise estates have freezes, audit windows, and month-end. Red team that ignores the calendar creates political damage:
- Schedule campaigns in published windows.
- Avoid destructive tests near financial close for finance agents.
- Pre-brief SOC so alerts from test identities are not mishandled as real attackers without context — and still ensure real attackers cannot hide as "tests."
- Document test identity naming conventions in the IR runbook.
Authorization letters should list systems, dates, identities, and data classes. Verbal "go ahead" is not authorization.
Sample release gate checklist (copy into your wiki)
| Check | Owner | Status |
|---|---|---|
| Smoke goldens green in CI | Eng | |
| Full goldens within threshold | Eng | |
| Automated injection pack pass | Security | |
| No open Sev1/Sev2 from last campaign | Security | |
| Trace sampling works in pre-prod | SRE | |
| Kill switch drill less than 90 days old | Ops | |
| Risk sign-off on residual items | Risk acceptor |
Empty cells at go-live are a choice. Make it with a name and a date, or do not ship.
If you are facing this
If you are planning or scaling a Microsoft 365 Copilot / enterprise AI program and want a practitioner review of readiness, controls, metrics, or agent governance — get in touch. Bring inventory, residual risk, and a sponsor who can decide; we still take this work.