The root-cause analysis landed earlier this month, and it says what the crash dumps already said: a channel file carrying 21 input fields met sensor code expecting 20, in kernel mode, on several million machines at once. CrowdStrike has promised staged deployment for content updates, customer control over content timing, and more validation in the pipeline. Other endpoint vendors are making similar commitments, mostly unprompted, which tells you the whole category understood July as a warning shot rather than one company's bad day.
Vendor promises are half the fix, and the half we do not control. The other half is what an estate can engineer for itself, so that the next defective content update — from any vendor, because there will be one — lands as an inconvenience instead of a lost weekend. We have spent the past month running exactly this program with clients. None of it is exotic. Most of it is scheduling, delegation, and the willingness to rehearse.
Rings for content, because rings for binaries weren't enough
July's failure mode was not an unpatched fleet or a skipped test cycle on the customer side; it was a class of update that had no rings at all. Sensor builds were pinned N-1 almost everywhere we looked, and the content channel ignored that entirely. So the first engineering job is to give content the same shape every other change gets: a canary population, a soak period, promotion criteria, and a rollback answer.
The canary group needs design, not volunteers. One to five percent of the fleet, spanning every hardware model, every site type, every OS build, and every business function — with a hard rule that no member is a single point of failure. You want one machine from dispatch, not all of dispatch. Where your security vendor now exposes content-update policy, wire the rings into it. Where it does not, that gap goes on the contract agenda below, and in the meantime the same ring structure still pays for itself on Windows Update, agent versions, and configuration changes in Intune.
| Ring | Population | Soak | Promotion gate |
|---|---|---|---|
| Canary | 1–2 percent, engineered spread | 4 business hours | zero crash telemetry, health checks green |
| Early | 10 percent, one full site included | 24 hours | helpdesk ticket rate flat |
| Broad | remainder | rolling | automatic once Early holds |
Four hours of soak would not have saved July's canaries — the bad file was live for 78 minutes. It would have saved the other 98 percent of the fleet, and that is the entire point of rings.
Promotion should run on telemetry, not on a calendar. Crash counts from the canary population already sit in your endpoint analytics; wire the promotion gate to them, so that a soak window containing three unexplained bug checks holds the release without a human needing to be brave in a meeting. Rings that promote automatically on green data get used. Rings that require a decision get skipped the week everyone is busy, which is every week.
Drill the key path like a fire alarm
Escrow coverage is necessary and not sufficient. July's real test was retrieval at volume, at three in the morning, by tired people. Treat the key path like a fire drill, with a measured evacuation time.
The quarterly exercise we now run with clients: pick twenty random devices; have the helpdesk produce each recovery key through the delegated, audited path; have one actual end user pull their own key through the device list in the My Account portal; time everything. Minutes-to-key is the metric. Anything over five minutes for a technician, or any device whose key simply is not there, is a finding.
The gaps are boringly consistent. Machines imaged before the escrow policy existed. Devices that moved from AD join to Entra join and left their key behind. Delegation so tight that three people hold query rights for eight thousand machines. For the first two, push the escrow forward as a remediation script:
$vol = Get-BitLockerVolume -MountPoint "C:"
$vol.KeyProtector |
Where-Object KeyProtectorType -eq "RecoveryPassword" |
ForEach-Object {
BackupToAAD-BitLockerKeyProtector -MountPoint "C:" -KeyProtectorId $_.KeyProtectorId
}
For the third, build a surge role now: a group carrying key-read rights that can be populated with one approval during a declared incident, every read audited, membership emptied when the incident closes. Deciding this during the event is how recovery keys end up read over speakerphone to anyone who sounds sufficiently stressed.
Close the loop at both ends of the device lifecycle while you are in there. Imaging and Autopilot flows should verify escrow as a completion gate, so no machine enters service without its key on file. And rotation belongs in the drill: rotate twenty keys, confirm the replacements land in Entra ID, confirm the old ones stop working. July put thousands of recovery keys onto paper and phone bridges; the estates that rotated afterward closed that exposure, and the ones that rehearsed rotation beforehand closed it in an afternoon.
Recovery media that exists before it's needed
Microsoft improvised a bootable USB recovery tool at global scale inside a weekend, which was impressive and is not a strategy. Yours should be standing inventory. Per-site kits: two USB sticks with current WinPE plus the storage and network drivers for the hardware actually at that site, a laminated instruction card written for a stressed non-expert, and a named owner who refreshes the media quarterly. Sites without a kit borrow one at courier speed, which is to say too slowly.
Verify WinRE itself while you are in there. A surprising share of July's stubborn machines had a broken or absent recovery partition; reagentc /info answers in one line and collects fleet-wide as an Intune compliance script. A machine without working WinRE turns a three-minute repair into a reimage, and you would rather learn which machines those are from a report than from a technician at 2 a.m.
For campus and warehouse sites, network boot beats sneakernet: a PXE option carrying the same WinPE image turns per-machine USB shuffling into a per-VLAN operation. And print the per-model boot-menu keystrokes on the laminated card — nobody remembers which vendor wants which function key at three in the morning, and the person holding the USB stick may be a site manager rather than a technician.
Also decide the reimage threshold in daylight: the point at which you stop performing surgery and wipe the device. With Autopilot and decent application packaging, a rebuild frequently beats twenty-five minutes of manual recovery on an uncooperative laptop. Estates that knew their threshold in July routed machines decisively. Estates that did not ran triage debates at the deskside, per device, while the queue grew.
Cloud PCs and AVD as the spare tire
We operate virtual desktop estates north of ten thousand seats, so July handed us a controlled experiment. Session hosts and Cloud PCs crashed exactly like physical laptops — same agent, same kernel, no magic immunity. The difference was the recovery curve: golden images rolled back, host pools reprovisioned in bulk, restore points applied, and a user with a bricked physical laptop could still reach a Cloud PC from any browser that boots. Virtual estates measured recovery in hours; deskside estates measured it in days.
Two design consequences follow. First, virtual desktops earn a line in continuity planning as surge capacity: a pool you can expand when physical endpoints are degraded, with licensing understood in advance and capacity actually tested — a large spin-up is not instant, and regional capacity is not infinite on the worst morning of the year. Second, put your VDI and Cloud PC images in the earliest content ring. Fast rollback makes them the cheapest canaries you own: they can absorb a bad update at breakfast and be rebuilt by lunch.
Questions your next endpoint-agent contract should answer
July was partly a procurement failure — the industry's collectively, ours included. Controls we assumed existed did not, and nothing in the paperwork said they had to. Questions we are now writing into EDR and endpoint-agent agreements as terms rather than slideware:
- Does detection content deploy through staged rings, with configuration the customer can see and set?
- Can we pin or defer content updates, and at what granularity — fleet, group, or device?
- Is there a machine-readable feed of what content shipped when, so monitoring can correlate a push with a problem?
- What is the vendor's own canary process, against what hardware and OS matrix, and how long is the internal soak?
- What runs in kernel mode versus user mode today, and what is the roadmap for moving logic out of the kernel?
- What are the notification commitments — time to first customer notice — when a vendor-side defect impacts availability, and what does the liability clause actually pay?
No vendor will accept all of it. The exercise is discovering which ones accept none of it, before renewal rather than after the next incident.
Rehearse the day everything fails at once
Fleet-wide simultaneous failure used to be a tabletop hypothetical introduced with an apologetic cough. It is now a documented, dated precedent, which puts it on the exercise calendar next to the DR failover.
We ran one in early August for a 9,000-seat logistics firm. Inject: 05:40, 55 percent of Windows endpoints boot-looping, dispatch and cross-dock floors affected, cause unknown for the first hour. The findings were the value. The incident bridge number lived in a Teams channel nobody could open from a dead laptop. The group holding LAPS read rights was four people, all in one time zone. Recovery USB kits existed at two warehouses out of eleven. And nobody owned the authority to declare reimage-over-repair, so the exercise spent twenty minutes relitigating it — exactly as the real weekend would have. Every finding was fixed inside a month, for roughly the cost of the meeting that found it.
Run yours assuming the trigger is not the security agent next time: a bad Windows servicing update, a fleet-wide firmware defect, a certificate expiry that takes an application stack down. The muscle being trained is the same, and it is the muscle, not the scenario, that pays.
If you're facing this
If your July after-action list is still a list, the one-month mark is the honest deadline for turning it into rings, drills, and dated calendar entries. We have run this program across several estates since the outage and can compress the schedule considerably. Talk to us.