The demo always works. That is not a moral failing — demos are supposed to work. The failure mode we keep walking into is treating a successful demo as evidence the system is ready for tenants who will ask questions nobody put in the slide deck.
This note is the checklist Hyper Mind Tech uses before a retrieval-augmented generation feature leaves shadow mode. It is opinionated on purpose. If your organization can ship without these gates, you are either very small, very lucky, or about to learn why "we will add evals later" is a schedule fiction. Pair this with Azure OpenAI landing for regulated orgs when the platform path is Azure, and with AI app security review before GA when the app has side effects beyond answers.
Start with the failure modes, not the model
Before model selection, we write down three lists:
- Questions that must be answered from retrieved context (with citations the user can open).
- Questions the system must refuse (out of policy, out of corpus, or "we do not know").
- Actions that must never be taken by the model alone (refunds, account changes, destructive admin).
If list one is empty, you do not have a RAG problem — you have a chatbot problem, and the evaluation strategy is different. If list three is non-empty and you still plan a free-form agent with tools, stop and redesign the tool surface first — narrow tools and human gates before fluent autonomy.
Most product teams skip this because it feels like product management. It is. Shipping AI without it is how you get a fluent system that is wrong in expensive ways.
// Production RAG control loop
[User query] --> [Auth + tenant context]
[Auth] --> [Retriever + hard filters]
[Retriever] --> {score floor met?}
{yes} --> [Grounded generation + citations]
{no} --> [Refuse / escalate]
[Answer] --> [Eval scores + traces]
[Traces] --> [CI gates / on-call]
Retrieval contracts beat "more context"
We treat the retriever as a product surface with a contract:
- Corpus: which sources are in, which are out, who owns freshness.
- Chunking: size and overlap chosen for the document class, not a blog default.
- Filters: tenant isolation is non-negotiable — hard filters at the index layer, never "the prompt will respect tenancy."
- Top-k and score floors: what happens when nothing clears the floor (refuse or escalate, do not invent).
On Azure, that often means Azure AI Search with tenant-scoped filters and a clear story for document ACL sync. Postgres + pgvector is fine for smaller corpora if you are honest about operational load. The wrong choice is a single shared index with prompt instructions about "only use docs for this customer."
| Contract element | Minimum for pilot | Minimum for GA | Owner |
|---|---|---|---|
| Corpus inventory | Named sources + owners | Freshness SLAs + change process | Product + data |
| Tenant filter | Enforced in index query | Tested negative cases in CI | Eng + security |
| Score floor | Defined and logged | Tuned from shadow traffic | Eng |
| Citation policy | Required for factual answers | Broken citation = fail eval | Product |
| Refuse path | Implemented | Measured refusal quality | Eng + risk |
| Delete/tombstone | Manual process | Automated within SLA | Ops |
Goldens from the ugly tickets
A golden set built from happy-path docs will pass forever and tell you nothing. We build goldens from:
- Real support tickets (redacted), including the ones where the human agent was wrong once and corrected.
- Adversarial paraphrases of the same intent.
- "Almost in corpus" questions that should refuse.
- Multi-hop questions that require two chunks — if your product claims to handle them.
Size: start at 50–100 cases if you are a small team; grow to a few hundred before GA for a broad help surface. Label expected behaviour: answer + citation IDs, or refuse with a reason class. You do not need perfect natural-language references for every row; you need a judge that can score groundedness and refusal quality.
We use a mix of deterministic checks (citation IDs present and resolvable) and LLM-as-judge with a fixed rubric versioned in git. Human spot-checks on a sample every release — not because the judge is useless, but because judges drift when you change the judge model without noticing.
How we structure a golden row
| Field | Purpose | Failure if missing |
|---|---|---|
case_id | Stable identity across runs | Cannot trend regressions |
query | User-facing wording | Harness only tests lab English |
intent_class | Group for coverage reports | Blind spots hide in averages |
expected_citations | Doc/chunk IDs allowed | Groundedness becomes opinion |
expected_behavior | answer / refuse / escalate | No pass-fail contract |
severity_if_wrong | Sev1–Sev4 for triage | Everything is "interesting" |
source_ticket | Link to redacted origin | Goldens rot without ownership |
Regression gates in CI
If evals only run on a laptop the week before launch, they are theatre. Wire them into CI:
- PR checks on a smoke golden (fast, ~20 cases).
- Nightly on the full set.
- Block merge or block the release pipeline when groundedness or refusal rates drop past agreed thresholds.
Store traces: prompt version, retrieval query, chunks returned, model, latency, cost. OpenTelemetry into something you will actually open (Langfuse, Arize, or your existing APM). When a customer reports a bad answer, you need the path from request ID to chunks in minutes, not a Slack archaeology expedition.
// Eval gate stages
[PR] --> [Smoke goldens ~20]
[Smoke pass] --> [Merge allowed]
[Nightly] --> [Full goldens]
[Full pass] --> [Release candidate]
[Shadow traffic] --> [Compare + drift report]
[Drift OK] --> [Feature flag on]
[Any gate fail] --> [Hold flag / fix / retest]
| Gate | Typical threshold (example) | Action on fail |
|---|---|---|
| Groundedness on goldens | Drop greater than 3 pts vs last release | Block release |
| Correct refuse rate | Below agreed floor on policy cases | Block release |
| Citation resolvability | Less than 98% of claimed IDs resolve | Block release |
| p95 latency | Exceeds SLA | Warn or block per product |
| Median cost per successful answer | Greater than 1.5× baseline | Warn; investigate retrieval bloat |
| Injection pack (if tools) | Any critical success | Block release |
Thresholds are examples — set numbers from your risk appetite, then defend them when someone wants to "just ship the model upgrade."
Shadow mode is not optional for customer-facing RAG
Ship behind a feature flag. Run production traffic through the new path without showing answers, compare to the old path or to human baselines, and only then flip visibility. Shadow mode is where you learn about:
- Query distributions your golden set never saw.
- Latency tails under real load.
- Index freshness bugs (docs deleted in the CMS still serving).
Yes, it costs tokens. That cost is cheaper than a week of trust repair.
Shadow-mode operating checklist
- Sampling plan — percent of traffic, max daily spend, stop if cost spikes.
- Comparison metrics — agreement with human/old path, refuse rate, empty retrieval rate.
- Privacy — traces redacted; no long-lived storage of raw PII in eval stores.
- On-call — who watches the dashboard during the first two weeks of shadow.
- Exit criteria — numeric conditions for flag on, not "feels good."
- Rollback drill — practice turning the flag off before you need it.
Failure classes we score separately
Lumping all bad answers into one "quality" score hides the failures that matter. We score at least these classes:
| Failure class | Example | Typical fix |
|---|---|---|
| Ungrounded fluent | Confident answer with no support in chunks | Raise floor; tighten prompt; citation require |
| Citation theater | Links that do not support the claim | Deterministic citation checks |
| Wrong tenant leak | Content from another customer | Hard filters + negative tests |
| Missed refuse | Answered policy-blocked topic | Expand refuse goldens |
| Stale retrieval | Deleted or superseded doc still cited | Tombstones + freshness jobs |
| Over-refusal | Refused when corpus had the answer | Tune floor and refuse rubric |
| Latency-induced truncate | Partial answer under timeout | Retrieval budget + model choice |
Security-adjacent classes (injection, tool abuse) sit on top when the system has tools. Map language to the OWASP LLM enterprise threat model so risk committees hear a dialect they already approved.
Staffing and ownership that outlive the pilot
| Role | Owns | Anti-pattern |
|---|---|---|
| Product owner | Intent lists, severity, GA criteria | "Engineering will decide quality" |
| Retrieval engineer | Index, filters, chunking, freshness | "The model will compensate" |
| Eval owner | Goldens, CI gates, judge version | Evals as side project of one intern |
| On-call | Flag kill, incident traces | "Ping the vendor" as runbook |
| Security / risk | Tenancy tests, residual risk accept | Review only at launch day |
If eval owner and retrieval engineer are the same hero who is also on-call forever, you have a single point of failure with a GitHub handle.
What we refuse to call "done"
We will not call a RAG feature production-ready if any of these are true:
- No tenant isolation at the retrieval layer.
- No citation requirement for product answers that claim to be factual from your docs.
- No refuse path for empty retrieval.
- No versioned prompts and no eval history.
- No on-call owner who knows how to disable the flag.
- No shadow period with exit criteria written down.
- No residual risk acceptance for known goldens that still fail.
Model brand is the least interesting part of that list. Claude, GPT-4o, or an open model on Azure — pick for cost, latency, and policy. The system around the model is what fails in production.
Field vignette (anonymized)
A B2B SaaS team shipped a help RAG after a week of internal demos. Groundedness on their happy-path set was 94%. In week two of production, a multi-tenant filter regression returned chunks from a neighboring customer for three queries before anyone noticed — the answers looked "mostly right" until a support engineer recognized another logo in a citation title. The fix was not a better system prompt. It was hard filters, negative tenancy tests in CI, and a kill switch the on-call had never practiced until that week. They spent more on the incident response than they would have spent on a month of shadow mode.
Another team refused to block releases on eval drops because "the model upgrade is better overall." Overall groundedness rose two points; correct refuse on policy cases fell eight. Compliance found it before customers did. The release process now treats refuse rate as a hard gate equal to groundedness.
Sequencing with Microsoft 365 Copilot programs
Custom RAG and M365 Copilot are not substitutes. Copilot rides Graph permissions and seat economics; custom RAG is for productized answers over your corpus with your tenancy story. Do not skip evals on custom RAG because "we also bought Copilot seats." Seat programs still need permissions-first readiness; product RAG still needs golden sets and CI gates.
Closing stance
Evals before the feature flag is not academic purity. It is how you keep the right to run generative systems in front of customers. Build goldens from ugly tickets. Contract the retriever. Gate CI. Shadow before you show. Trace until on-call can sleep.
The organizations that treat RAG like software — with regression, ownership, and rollback — keep shipping. The ones that treat it like a demo that graduated on vibes learn the hard way that fluency is not truth.
If you are facing this
Bring the corpus shape, the tenancy model, and three examples of answers that would get you in trouble. We can tell you in a short written read whether you need a retrieval redesign, an eval harness, or both — and what a fixed-scope discovery would cover. Get in touch; no obligation to build with us after that.