At 3:07 on a Monday morning in October, a monitoring alert fired for a government client of ours: password hash synchronization heartbeat missing for 120 minutes. By 3:40 we knew the Azure AD Connect server had come back from its patch-weekend reboot with the sync service in a failed state. By 6:00 it was fixed — and nobody in the organization ever knew, which is the entire point of this post.
Here is the uncomfortable arithmetic of hybrid identity in late 2020. This year moved everyone's work into Office 365, Teams, and whatever sits behind Azure AD. Which means the sync engine and the authentication path from your on-prem directory to the cloud is no longer plumbing — it is the thing standing between eight thousand people and their Monday. If Azure AD Connect stops, new hires do not appear, password changes do not propagate, and deprovisioned accounts keep working. If your pass-through authentication agents stop, nobody signs in at all. We have watched both happen this year, across our own managed estates and in the wreckage of a few first calls.
And yet in most organizations the AD Connect box is still treated like a utility server: patched with the general population, monitored for ping and disk, owned by whoever set it up in 2017. We run it as tier-0 — the same tier as domain controllers — and this post is the operational model, learned the way these things are always learned.
Hybrid identity is tier-0, treat it accordingly
Tier-0 means three concrete commitments, not a label:
- Tier-0 security. The AD Connect server holds credentials and cryptographic material that can impersonate your directory. It gets domain-controller-grade hardening: no browsing, no shared local admins, restricted logon, its own patching ring, and it lives in the same protection scope as your DCs. An attacker on your sync server is an attacker in your directory — this is not a theoretical chain, and everything we have seen this year says federation and sync infrastructure is exactly where sophisticated actors look.
- Tier-0 availability. Sync outages are silent. Mail flows, tokens refresh, and everything looks fine — until the new-starter accounts are missing, the terminated contractor still has access, or a password change strands a user between two credential states. The blast radius grows with every hour undetected.
- Tier-0 ownership. A named owner, a runbook, and a tested recovery procedure. "The person who set it up" is not an owner if they left in 2019, which in our experience they did.
The failure modes we have actually been paged for
Four years of running hybrid identity across government and enterprise estates gives us a short, boring, painfully consistent list:
| Failure | Symptom | Detection window that matters |
|---|---|---|
| Sync service down after patching | No exports; heartbeat stops | Under 2 hours |
| AD DS connector account locked or password reset by a helpful admin | Import errors on the on-prem connector | Under 2 hours |
| Export errors accumulating (duplicate proxyAddresses, UPN clashes) | Specific objects never sync; users report selectively | Daily review |
| Certificate or TLS change breaks the outbound channel | Sync or PTA agents cannot reach Azure AD | Immediate |
| Auto-upgrade lands a build with a regression | Subtle rule changes, unexpected attribute flows | Weekly review |
| Well-meaning engineer runs a full sync with a bad OU filter | Mass deletions pending | Immediate, before export |
That last one deserves its own paragraph, because it is the scariest and the most preventable. Azure AD Connect ships with an accidental-deletion threshold — export deletion prevention, default 500 objects — and we have twice seen it save an estate: once from a broken OU filter, once from a domain controller that answered an import with a partial view of the directory. Know your threshold, size it to your org, and treat a tripped threshold as a stop-everything event, not an obstacle to click through:
Get-ADSyncExportDeletionThreshold
# and when you must investigate a stuck cycle:
Get-ADSyncScheduler
Get-ADSyncConnectorRunStatus
If you take one operational habit from this section: nobody runs Start-ADSyncSyncCycle -PolicyType Initial on a production estate without a second person and a stated reason. Full syncs after rule changes are routine; full syncs as a troubleshooting reflex are how mass deletions happen.
The staging server is the cheapest insurance you will ever buy
Azure AD Connect supports exactly one active sync server per tenant — but it supports any number of servers in staging mode, importing and computing but never exporting. Every estate we run has one, and we are consistently surprised how many estates we inherit do not.
The staging server buys you three things. First, failover: if the primary dies, you flip the staging server to active in minutes — a configuration change, not a rebuild — instead of reinstalling and re-deriving four years of undocumented sync rule customizations under pressure. Second, a rehearsal environment: upgrades and rule changes land on staging first, and the built-in CSExport/CSAnalyzer comparison shows you exactly what would have changed before it does. Third, documentation-by-existence: keeping staging in sync forces the custom rules to live somewhere reproducible, exported and version-controlled, rather than only in the memory of the primary server.
One warning from the field: a staging server left unpatched and unmonitored for a year is not insurance, it is a second incident waiting behind the first. It follows the same tier-0 regime as the active server. Both of them, ideally, on separate hosts, separate patch nights, and — after this year's Hyper-V cluster incident at one client — separate failure domains.
PTA agents: three is the floor, placement is the point
For estates on pass-through authentication (we still put plenty of clients there; our 2017 decision tree on PTA versus ADFS has aged well), the authentication agents are the sign-in path, full stop. If every agent is unreachable, authentication fails for everyone — password hash sync can be your fallback only if you deliberately keep it enabled alongside PTA, which we do, on every estate, without exception. Synced hashes as a break-glass authentication method has rescued two of our clients this year; the objections to it are almost always philosophical, and the outages are not.
Agent math and placement, as we deploy it:
- Three agents minimum, sized so that any one can carry peak sign-in load alone. They are lightweight; the constraint is never CPU, it is placement.
- Separate failure domains. Not three VMs on one cluster, not three servers behind one proxy, not three boxes in one rack. One client's "redundant" agents all died together in a proxy appliance change because every agent's outbound path went through it — the sign-ins failed over to nothing.
- Watch the outbound dependencies. Agents make persistent outbound connections to Azure; TLS inspection appliances, proxy auth changes, and firewall "cleanups" are the leading killers. Every agent's egress path is documented in the runbook, and the firewall change board knows those rules are tier-0.
- The AD Connect server's own agent does not count toward your three. It is a convenience install, co-resident with your other single point of failure.
Monitoring sync and auth like the tier-0 service it is
Azure AD Connect Health is the obvious first layer and we deploy it everywhere the licensing allows: sync error reporting, heartbeat, agent status for PTA, and — for the estates still on ADFS — the sign-in telemetry that has flagged two brute-force campaigns for us this year. But Connect Health alerts by email, on portal timelines, and email at 3 a.m. wakes nobody.
So we wire the pipeline into the same paging path as domain controllers. The signals that have earned their place:
- Heartbeat and export freshness. A sync cycle that has not exported in 90 minutes pages. (Default cycle is 30; we alert at three missed.)
- Export error count, thresholded and trended. Ten new duplicate-attribute errors after an HR import is a story worth reading same-day.
- PTA agent count. Fewer than the deployed number for more than 15 minutes pages, even at full service — you are one failure from discovering the redundancy was theoretical.
- Service and scheduler state on the box itself, via the same SCOM infrastructure that watches everything else.
Get-ADSyncScheduleroutput has a SyncCycleEnabled property; false in production is a page, not a curiosity. We have found it disabled twice — both times a troubleshooting session from months earlier that never got re-enabled.
The runbook that saved a Monday
Back to that 3:07 alert. The reason the October incident cost one engineer three hours instead of costing a client their Monday is entirely unglamorous: a runbook, printed and current, that said what to check, in what order, and what "known good" looks like. Service state, scheduler state, connector run history, last export timestamp, deletion threshold status, and the decision points — including the pre-authorized criteria for flipping the staging server to active without waking a manager.
The incident itself was nothing: a patch reboot left the service in a bad state and a dependent service timing issue kept it there. The value was that at 3 a.m., nobody had to be smart. They had to be literate. That is what tier-0 operations means in practice — engineering the need for heroics out of the system, one boring document at a time.
If you're facing this
If your Azure AD Connect server is a pet, your PTA agents share a failure domain, or nobody has tested the staging failover since it was built, you are carrying a silent tier-0 risk that this year has made everyone else's problem too. We run hybrid identity health checks that take under a week and end with a runbook, a monitoring config, and a short list ranked by blast radius. Get in touch — ideally before your own 3 a.m. page does the assessment for you.