There are two kinds of SCOM deployments: the ones where the operations team trusts the console, and the ones where 4,000 unread alerts sit in the inbox rule everyone created in month two. The product is the same in both. The difference is tuning discipline, and almost nobody writes about it because tuning is unglamorous, continuous work rather than a project with a go-live date.
We inherited the second kind of deployment last year: a 400-plus server estate we had virtualized onto Hyper-V with full System Center — SCVMM for fabric, SCCM for configuration, SCOM for monitoring. SCOM was installed, management packs were imported enthusiastically, and then reality arrived: 700 alerts a day, of which perhaps a dozen described anything a human should act on. The night shift had learned to ignore the console, which meant the estate was effectively unmonitored with extra steps. Six months later the same environment generates under 40 alerts a day, and when something pages at 3 a.m. it is real.
This post is the record of how we got from one state to the other. None of it is clever. All of it is repeatable.
Alert fatigue is a management decision, not a product flaw
The root cause was visible in the management pack list: 60-plus packs imported, most with default thresholds, several monitoring products the estate did not meaningfully run. Every management pack author has an incentive to monitor everything their product can express; none of them know your environment. Importing a pack without a tuning pass is choosing to receive the vendor's maximum possible alert volume.
Our operating rule since: a management pack earns its way in. Before import, we answer three questions in writing — which classes will it discover, which monitors do we actually want, and who acts on its alerts? If the third question has no name attached, the pack does not go in. On this estate we removed eleven packs entirely (including a network-device pack monitoring switches the network team already watched with different tooling — duplicate alerting is worse than none, because it trains people that alerts are somebody else's problem).
The second structural fix: override management packs, one per vendor pack. All tuning lives in dedicated unsealed MPs — HMT.Overrides.SQL, HMT.Overrides.Windows.OS, and so on — never in the Default Management Pack. This is old guidance and still routinely ignored. When overrides are scattered through the default MP you cannot version them, export them, or reason about them, and every SCOM migration becomes archaeology.
The weekly tuning ritual: top talkers, every Tuesday
Tuning is not a phase; it is a standing 45-minute meeting. Every Tuesday we pull the noisiest alerts of the trailing week — a simple data warehouse query, or quick-and-dirty from the operational database:
Get-SCOMAlert -ResolutionState 0 |
Group-Object Name |
Sort-Object Count -Descending |
Select-Object -First 20 Count, Name
For each of the top 20, one of four verdicts:
- Real and actionable — keep it, and make sure the alert description tells the on-call engineer what to do. If we cannot write the action, it is not actionable.
- Real but wrong threshold — override the threshold for the class or group. Logical disk free space is the classic: 10 percent free on a 4 TB CSV is 400 GB, which is not an emergency. We override to absolute-value thinking per disk class.
- Real but nobody will ever act — disable the monitor. This takes nerve the first few times. Informational noise about a service that flapped once and recovered is not monitoring; it is journaling.
- Symptom of a deeper misconfiguration — the valuable category. A recurring Hyper-V dynamic memory pressure alert on the same three hosts was not an alerting problem; it was a capacity problem our density math had missed after Meltdown/Spectre patching changed our overhead assumptions.
Verdicts get recorded — date, alert, decision, override MP — in a plain change log. In week one of the ritual we cut daily volume roughly in half. The curve flattens after two months, but it never reaches zero, because the estate changes underneath you: every new SQL instance, every OS upgrade cohort re-opens a tuning conversation.
One trap worth naming: do not tune by turning entire monitors off when the real fix is aiming them. SCOM groups are the tool. We maintain groups like "SQL Production", "SQL Dev/Test", and tier the thresholds — dev boxes warn where production pages. Same monitor, different posture, expressed in overrides scoped to groups.
Maintenance mode automation, or: stop paging yourself about patching
The single largest source of false alerts in this estate was us. Every SCCM patch window produced a blizzard of heartbeat failures, service stops, and reboot alerts — all self-inflicted, all training the on-call rotation to ignore the console during exactly the windows when real failures hide among planned ones.
The fix is maintenance mode, and the fix for maintenance mode is automation, because "the engineer remembers to set it" is not a control. We wired it to the patching process itself: the SCCM task sequence's first step calls a small script that drops the machine into SCOM maintenance mode for the expected window, and the last step ends it early on success.
$instance = Get-SCOMClassInstance -Name "$env:COMPUTERNAME.$env:USERDNSDOMAIN"
Start-SCOMMaintenanceMode -Instance $instance -EndTime (Get-Date).AddHours(3) `
-Reason PlannedOther -Comment "SCCM patch window"
Two refinements from production. First, put the cluster objects into maintenance mode when patching cluster nodes, not just the node — otherwise the cluster pack raises failover alerts that are, again, you. Second, alert on maintenance mode abuse: a report of objects in maintenance mode longer than 24 hours catches the server someone silenced in November and forgot. We found four of those during the initial cleanup. One of them had a failed backup agent the whole time.
The 20 alerts that matter
After tuning, we sat down with the operations lead and wrote the actual contract: which alerts page a human immediately, at any hour. The list surprised everyone by being short. For this Hyper-V estate it is about 20 alert types across five families:
- Fabric: cluster node down, CSV entering redirected access, storage latency sustained above threshold, Hyper-V host heartbeat failure.
- Capacity cliffs: CSV free space below the runway needed to absorb one node's VMs, memory pressure preventing VM starts.
- Tier-0 dependencies: domain controller unreachable, DNS resolution failures, certificate expiry inside 14 days on the systems that terminate anything.
- Data protection: backup job failures on the second consecutive occurrence — one failure self-heals often enough that we page on the pattern, not the event.
- The application tier's own short list, negotiated with each app owner, on the same "if this fires, someone acts" standard.
Everything else flows to the console and the daily review, not to phones. Writing the page-worthy list down did something subtle and valuable: it converted monitoring from an ambient anxiety into an explicit service level. When an incident occurs that the list would not have caught, we amend the list — that is the post-mortem habit, and it is how the list stays honest instead of growing back into noise.
The data warehouse is the quiet payoff
Alerting gets the attention, but six months in, the SCOM data warehouse has become the estate's capacity-planning source of truth. Performance collection rules — tuned down to the counters we actually chart — feed quarterly reviews: CPU-ready-equivalent pressure per host, CSV IOPS and latency trends, memory demand curves per cluster. When we re-ran the density math after the Spectre-mitigation overhead landed, the answer came from SCOM reporting, not from spreadsheets and vibes. Same story for the annual budget: "these two clusters cross 80 percent sustained memory demand in roughly February 2020 at current growth" is a sentence that gets a purchase order approved.
If you are only using SCOM for alerts, you are paying for a database and using it as a pager. Tune the collection rules (default packs collect far more than you will chart — that is disk space and aggregation time), then build the five reports your capacity conversation needs.
If you're facing this
If your SCOM console has become the thing everyone filters to a folder, the platform is not the problem — the tuning contract is missing. We have run this cleanup on estates of 400-plus servers and left behind a weekly ritual the in-house team actually keeps. Get in touch if you want the noisy-to-trusted playbook run against your environment.