In the span of five days, OpenAI shipped o1-preview — models that spend tokens reasoning before they answer — and Microsoft announced Copilot Wave 2, with Copilot Pages, agents surfacing in Business Chat, and an agent builder in Copilot Studio. The Wave 2 announcement was yesterday. Our inbox has responded accordingly: four separate clients this week asking some version of what is our agent strategy.
Fair question, wrong starting point. We have spent 2024 building and reviewing systems that get called agents, and the honest field report is that the word currently describes a demo aesthetic more than an architecture. The systems that survive contact with production look remarkably alike, and they look very little like the demos.
Terms first, because the vendors will not settle them for us. An agent is software that uses a model to decide which actions to take next — observe, plan, call a tool, repeat. The interesting variable is how much of the control flow the model owns. In the deployments that work, the answer is: as little as possible.
The week's announcements, read quickly
o1-preview is genuinely different. It reasons in hidden tokens before answering, and the gains on math, code, and multi-step planning are real, not benchmark theater. It is also slow, priced like it knows it, and the API arrived without system messages, streaming, or function calling — which tells you how early this is. Better planners will eventually make agent loops more capable. They will not fix a single item on the failure list below, because none of those failures are model-quality failures. Where o1-preview fits an agent program today is offline: as the planner you benchmark against in evaluation runs, or as the model that generates and critiques test cases for your golden set. At current latency and price, putting it in the hot path of an interactive workflow is a decision your users will report to you within the hour.
Wave 2 is organizationally more consequential. Copilot Studio puts agent building in front of business users, and SharePoint agents let a site become a conversational interface over its own content. If your Power Platform governance story took three years to catch up with citizen developers, note that the sequel has tool access. We are one day in; we have read documentation, not deployed, and a longer opinion today would be theater. But nothing about the governance posture below changes based on which builder wins, which is exactly why it is the place to start.
What ships and works: mostly pipelines
Strip the branding from the production systems we have seen succeed this year and you find a workflow engine. Deterministic control flow, written by an engineer who can be paged. One or two LLM steps doing the genuinely fuzzy work — extracting structure from a messy document, classifying a case, drafting prose for a human to edit. Tools with narrow scopes. Logging on everything. The model is an employee with a tightly written job description, not a founder with a vision.
On frameworks: LangGraph maps closest to how we think — an explicit state graph where the LLM nodes are visible and bounded, and where you can point at the arrow a bad decision traveled down. AutoGen is productive for research and multi-agent experiments; we have yet to see it carry a production workload we would defend in an audit. Semantic Kernel is the natural home for .NET shops that want plugins and planners inside existing services and existing CI. Prompt flow, inside Azure AI Studio, earns its keep less as orchestration and more as the harness that runs scored evaluations on a schedule.
But the framework argument is mostly displacement activity. The load-bearing question is simpler: for each step, does code decide what happens next, or does the model? Every notch of authority handed to the model buys flexibility and costs predictability, debuggability, and sleep. Spend it only where deterministic code demonstrably cannot do the job, and write down where you spent it.
There are places where fuller autonomy is defensible now, and they share a shape: internal-facing, low blast radius, reversible, cheap to verify. Enriching IT tickets with context before a human reads them. Drafting knowledge-base articles from closed incidents. Proposing — not applying — remediation steps. If a wrong action costs a shrug and a click, autonomy is fine. If a wrong action needs an apology, it gets a gate.
The four ways they break
Unbounded tool access. An agent holding a Graph credential that can read every mailbox for context does not have a prompt-engineering problem; it has a blast-radius problem. The token is the attack surface — prompt injection merely steers it. Dedicated app registrations per tool, least-privilege scopes, read-only until a reviewed case exists for write, managed identities so there is no secret to leak into a repo. And assume hostile input: the PDF resume that politely instructs the model to forward the mailbox somewhere interesting is no longer a thought experiment. Anything crossing the trust boundary gets treated the way we treat user input everywhere else — constrained, sanitized, and never allowed to redefine the task.
No eval harness. Someone improves a prompt on Thursday. A behavior three steps downstream regresses on Friday. Nobody notices until a customer does, and by then there have been four more prompt improvements. Without a golden dataset and a scored run gating every change, you are not operating a system — you are vibing in production. Evals are to agent systems what unit tests are to code, with the added twist that the platform drifts underneath you even when you change nothing.
No human gate on consequential actions. Sending external email, changing ticket states, modifying configuration, moving money: draft-for-approval is the default posture until months of eval and audit data argue otherwise, action by action. The pattern costs a click. It buys you an incident report you never have to write.
Cost blowups. Retry loops with large contexts are the classic: a failure mode where the system is down and billing beautifully. We reviewed a pilot this summer that spent a five-figure monthly budget in one weekend, retrying a malformed tool call with sixty thousand tokens of context attached to every attempt. Per-run token ceilings, per-day budget alarms, and a circuit breaker that stops the loop rather than logging it politely.
A claims pipeline with no drama in it
A mid-market specialty insurer came to us in the spring asking for an autonomous claims agent. What went live in August is a pipeline, and everyone is happier for it. Intake is deterministic. One LLM step extracts first-notice-of-loss emails and attachments into a typed schema. A second classifies severity, proposes routing, and drafts an acknowledgment. Above a severity threshold, a human adjuster approves the routing before anything moves; below it, straight-through processing with random sampling. Every model input, output, and tool call is logged and retained.
The eval set is four hundred adjudicated historical claims. Extraction accuracy and routing agreement are scored on every prompt or model change, and the weekly regression run has already caught two silent drifts — one following a model version update nobody on either team initiated.
Eval run 2024-09-13 (golden set: 400 claims)
extraction_accuracy 0.96 gate 0.95 PASS
routing_agreement 0.91 gate 0.90 PASS
tokens_per_claim p95 9,800 gate 12,000 PASS
latency_seconds p95 41 gate 60 PASS
First-touch time on new claims went from twenty-six hours to under two. Adjusters overrule the proposed routing about one time in eleven, which is precisely why the approval step is staying. Running cost lands around eleven cents of model spend per claim at current prices, and the weekly regression run costs roughly what one adjuster-hour does — numbers worth publishing internally, because the fastest way to kill an AI initiative is to let Finance discover the bill before anyone has shown it next to the saved hours. Nobody at the client calls the thing an agent. It does not appear to mind.
The gate we run before production
Before anything with a model in its loop touches a production system, we require all of the following, in writing:
- A named owner and an on-call path, because these systems fail at 2 a.m. like everything else.
- A tool inventory with each credential's scope justified individually. No shared super-token, ever.
- An eval harness with a golden set, scored gates, and a regression run wired into the release process.
- Consequential actions enumerated, each with its human gate or its documented, dated exemption.
- Per-run token ceilings and per-day budget alarms, with hard stops rather than notifications.
- Full trajectory logging — prompts, tool calls, outputs — with a retention answer Legal has seen.
- A kill switch that works without a code deploy, tested once.
- A data-boundary review: what leaves the tenant, and whether sensitivity labels are honored on the way.
- A rollback story no more exotic than: turn it off, the humans resume.
Clients occasionally push back that the gate is heavier than the agent. Correct — that is the test working. If a workload cannot justify the gate, it is a fine candidate for Copilot in an app someone already governs, and a poor candidate for autonomy. And when the Copilot Studio wave arrives and agent creation moves to the business, this list does not shrink; it becomes the intake form. Teams that treat it that way discover a quiet benefit: a gate that is public, written, and applied evenly turns a political argument about whose agent gets to ship into an engineering conversation about which requirements remain unmet. That trade is worth the paperwork on its own.
If you're facing this
If there is an agent initiative on your roadmap — or already running with a credential nobody has audited — the distance between demo and operations is where we spend our days. We are glad to walk the gate list against your specific case and tell you honestly which side of it you are on. Write to us.