December is when we write the honest retrospective, and this year the retrospective belongs to virtualization. Across several client programs — the largest a government estate, the next a healthcare group — we have now moved more than 400 physical servers onto Hyper-V, managed end to end with System Center: SCVMM for the fabric, SCOM watching it, SCCM keeping guests patched. The headline numbers our clients quote are around 50% infrastructure utilization improvement and, across the broader consolidation programs these P2V efforts anchor, roughly 35% infrastructure cost reduction.
Those numbers are real, and this post is about where they actually came from — because it was not where the business case slides said, and the gap between the two is full of transferable lessons about a server virtualization project at scale.
It is also about the servers we refused to virtualize. Every virtualization program is judged by its completion percentage, which creates a quiet incentive to force the last awkward workloads onto the hypervisor. Some of our best decisions this year were refusals.
Triage before tooling: the four-bucket sort
The instinct at project kickoff is to argue about conversion tools. Resist it. The tool handles the middle 80% of servers regardless; the project is won or lost on how you sort the other 20% before anyone converts anything. Every candidate server went through a four-bucket triage built from a discovery dataset — MAP Toolkit inventory plus 30 days of perf counters, because a server's reputation and its measured behavior are frequently different servers:
- Convert. Healthy OS (2008 R2 or later, plus a tail of stubborn 2003 boxes — more below), no exotic hardware dependencies, sane disk layout. Roughly 70% of the estate.
- Rebuild instead. The OS is old, the app is installable, and dragging eight years of sediment onto new infrastructure is a disservice. Fresh VM, reinstall, migrate data. Slower per server, but these VMs generated a fraction of the post-migration tickets the converted ones did. About 15%, and in hindsight this bucket should have been bigger.
- Retire. Servers whose function had quietly ended: the print server for a closed office, the "temporary" FTP box from 2011, three servers nobody could attribute until we powered them off with a two-week rollback window and waited for the phone to ring (it rang once). Nearly 10% of the physical estate simply did not need to exist — the cheapest consolidation there is.
- Refuse. The remainder — kept physical on purpose, covered below.
One operational note on the 2003 stragglers: out of support since last year, and every one of them went to the rebuild or retire buckets. Virtualizing an unsupported OS preserves the problem in amber. We declined to do it, in writing, every time a project manager suggested it would be faster.
The conversion machinery, without SCVMM's help
An awkward era fact: SCVMM dropped its built-in P2V capability after 2012 SP1, so the flagship management tool for the destination platform no longer converts sources. Our working answer this year was the Microsoft Virtual Machine Converter for the straightforward cases and Disk2vhd for the awkward ones — the servers with agents, filters, and drivers hostile to a live conversion — combined with a strict pre-conversion checklist: uninstall vendor hardware agents before converting (ghost NIC drivers and RAID management agents caused more post-conversion instability than everything else combined), document static IPs (converted machines present new NICs and cheerfully forget their addresses), and disable application services for a consistent capture.
Cutover discipline mattered more than tooling. Every conversion ran with a stated maintenance window, a verification script per server (services up, shares present, app-owner smoke test), and a rollback rule: the physical box stays racked, cabled, and disabled — not wiped — for two weeks. We rolled back four servers out of hundreds. All four rollbacks were boring, because the rule existed. The one genuinely bad night of the program was self-inflicted and instructive: a database server converted successfully but came up with its application unable to authenticate, because the cutover ran the same night as a scheduled password change on a service account — two changes, one window, classic. Since that night, our change calendar review is part of the conversion checklist itself.
Where the 50% actually came from
The business case attributed the utilization gain to "consolidation," full stop. The measured reality decomposed differently, and the decomposition matters because it tells you what to do on the next program:
| Source of gain | Share of improvement | The honest note |
|---|---|---|
| Raw consolidation (many idle boxes, few hosts) | About half | Most physical servers idled below 10% CPU; this part is real but was always the easy part |
| Retirement bucket | Meaningful minority | Servers that stopped existing improve every ratio; nobody's slide predicted it |
| Right-sizing at conversion | The sleeper | Physical specs were purchase-order archaeology; VMs got sized to measured demand |
| Dynamic Memory and density tuning | The remainder | Post-migration tuning, six months of SCOM data, unglamorous |
Right-sizing deserves the emphasis. A physical server bought in 2010 was specified for a budget cycle, not a workload; we routinely replaced 32 GB physical boxes with 8 GB VMs that never paged. But sizing to measured demand only works if you keep measuring — SCOM capacity reporting turned right-sizing from a one-time event into a quarterly practice, and the second-pass adjustments (both directions; we under-sized a few) are where the last chunk of the gain lives.
The cost reduction side — the number that reaches CFOs — came from hardware refresh avoidance first, datacenter footprint second (racks, power, cooling contracts), and operational simplification third. What did not shrink: licensing (Datacenter edition on the hosts is the right answer, but it is not free), storage (VMs sprawl faster than physical servers ever did, because provisioning is now minutes), and backup, which had to be re-architected rather than merely repointed.
Cluster sizing and the operational spine
The compute fabric ended up as several failover clusters of six to eight nodes rather than fewer giant ones — bounded failure domains, patching windows that fit in an evening, and Cluster-Aware Updating that operators actually trust. Sizing rules we now treat as house standards: N+1 capacity honestly calculated (the cluster must carry full load with a node down during a patch window, which is N+1 measured at the worst hour, not the average one); CSV layout planned per cluster with IO characteristics separated; and live migration networks sized generously because every operational convenience in the platform — patching, rebalancing, evacuating a sick host — is ultimately a live migration.
The other spine is management, and this is the part mid-sized estates most often skip: the platform is only as good as the System Center telling you the truth about it. SCOM with tuned management packs (tuned meaning: every alert that fires gets acted on — we cut the default alert volume dramatically before operators would trust it), SCVMM as the single point of fabric truth, SCCM patching guests on rings. Estates that virtualize without the management investment end up with 400 servers they can see less clearly than when the servers blinked at them from racks.
The servers we refused
The refusals, and the reasoning, because this list is more reusable than any success story: the fax server and its telephony board (physical hardware dependencies with no passthrough story worth the fragility); two applications licensed with hardware dongles whose vendors would not bless USB-over-IP; a vendor-managed clinical system at the healthcare client whose support contract said physical, full stop — virtualize it and every future support call starts with an argument; one enormous SQL Server whose measured IO the shared storage design could have carried only by degrading everyone else (it got new physical hardware and a line item, not an apology); and the PKI offline root, which stays offline and physical as a matter of doctrine, not capacity.
The pattern: we refused for hardware dependency, for support-contract reality, for measured performance economics, and for security doctrine. We never refused out of vague unease, and we never converted over a written vendor position. Both halves of that sentence took discipline.
If you're facing this
If a virtualization or consolidation program is on your 2017 roadmap — or you ran one and the utilization numbers never materialized — the levers are triage rigor, right-sizing against measurement, cluster design that survives patch night, and knowing what to leave physical. We have done this at the 400-server scale with the System Center stack around it; get in touch and we will walk you through the checklists, including the four-bucket sort and the refusal list.