We closed out a nine-month engagement this month building cross-site vSphere replication disaster recovery for a regional financial-services firm running two datacenters — a primary near Miami and a secondary three states north, far enough away to matter for a hurricane season neither of us enjoys living through. The client had backup. It did not have DR, and did not fully believe the difference until we made it prove one. This post is our field notes on what actually goes into a DR runbook that survives contact with a real failover, not a tabletop exercise nobody photographs.
Two sites, both on vSphere 6.5 — vCenter 6.5, ESXi 6.5 — was the easy part of the pitch. Host-based vSphere Replication doing the copying, Site Recovery Manager 6.5 orchestrating the failover: the SKUs were never the argument. The argument was RPO honesty, because the client's existing "DR plan" promised recovery numbers nobody had tested, for workloads nobody had prioritized, at a recovery site nobody had actually powered on in anger.
We've run DR and business-continuity programs for government and enterprise clients for years, across estates from a few dozen VMs to distributed environments past 700 servers, and the pattern repeats: the backup product is licensed and reported on religiously, while disaster recovery — a workload coming up on different hardware, on a different network, with its dependencies actually reachable — has never been rehearsed end to end. Here's what we built instead, including the test failover that found a single point of failure nobody had listed.
Backup is not disaster recovery
Say this to a steering committee and watch who nods too quickly. Backup answers "can we restore a file, a VM, a database to a point in time." DR answers "can the business run from a different site while the primary is gone." They share a word — recovery — and almost nothing else.
The estates that conflate them share a signature: nightly backup jobs with green checkmarks, a retention policy someone can recite, and a "DR site" that is really a backup target with some spare hardware nearby. Ask the harder questions and the checkmarks stop mattering:
- Can that backup restore to different hardware, on a different network, with different IPs? Most backup-and-restore runbooks assume you're restoring in place. DR assumes the place is gone.
- Does anyone know the order services have to come up in? A restored domain controller that boots after the application server it authenticates against is an outage with extra steps.
- Has anyone actually tested it, or does the plan just say restore the VMs and start them? That sentence is not a plan. It's a wish.
Our client had fourteen months of clean backup reports and zero minutes of tested failover. That gap is normal — we see it in most estates before we touch them — and it's the gap this engagement was hired to close.
vSphere replication disaster recovery: an honest RPO and RTO
Every DR conversation should start with numbers the business actually agreed to, in writing, per workload — not an aspirational "near zero" that nobody costed out. vSphere Replication gives you per-VM RPOs from 5 minutes up to 24 hours; Site Recovery Manager gives you an RTO that is mostly a function of how well your recovery plan is built, not the software. We forced the tiering conversation early, because the honest answer is that not everything gets a 5-minute RPO, and pretending otherwise is how DR programs die on the invoice.
The tiers we landed on, after three rounds of arguing with application owners who all believed their system was tier one:
| Tier | RPO | RTO target | What lived here |
|---|---|---|---|
| Tier 1 | 5-15 minutes | Under 1 hour | Core banking app, AD/DNS, the SQL cluster behind it |
| Tier 2 | 1 hour | 2-4 hours | Email connectors, internal line-of-business apps |
| Tier 3 | 4-24 hours | Next business day | File services, internal reporting, print |
| Not replicated | n/a | Rebuild from backup | Test/dev, anything the business agreed was disposable |
The friction isn't technical — vSphere Replication will happily give most VMs a 5-minute RPO if the WAN link and recovery-site storage can carry it. The real cost is sizing bandwidth and recovery-site compute for everything, and somebody signing off on that bill. Tiering is a budget conversation wearing an engineering costume, and saying that sentence out loud early is what keeps the project short.
Replication topology: stretched L2 versus re-IP at the recovery site
The network design decision that determines almost everything downstream: does the recovery site share a stretched Layer 2 with the primary, or does it re-IP on failover?
Stretched L2 is the smoother failover — a VM comes up at the recovery site with the same IP it had at home, and nothing downstream needs to know anything moved. It also costs real complexity: Layer 2 extension between sites, or NSX if you've made that investment (we kept it optional here, since the client hadn't deployed it and we weren't making NSX adoption a dependency of a DR project), and a failure domain that now spans both datacenters. Stretch the network and you can propagate a Layer 2 problem across sites just as efficiently as you avoid a re-IP.
Re-IP is what most of our clients actually run, this one included: recovery-site subnets are pre-provisioned and distinct from production, and Site Recovery Manager's IP customization handles the addressing swap. It moves the complexity elsewhere — every recovery plan needs accurate IP mapping per VM, and any application that hardcodes an IP instead of a name becomes a ticket you find during a test, not before.
Either way, what actually breaks failovers isn't the workload VM — it's what the workload VM assumes is already there:
- Active Directory and DNS have to exist at the recovery site before anything else recovers, or every subsequent VM boots into a network it can't authenticate against or resolve.
- Recovery plan priority has to encode that dependency explicitly. SRM groups VMs into priority groups with a boot order; if domain controllers and DNS aren't priority one, everything else is guessing.
- Re-IP mapping has to be complete, not just the subnets everyone remembered. This is the part that bit us, below.
Recovery plans as a DR runbook in Site Recovery Manager
A recovery plan in SRM is the closest thing to an actual DR runbook that survives being handed to someone other than the person who built it. Protection groups define what replicates together; the recovery plan defines order, dependencies, and the callouts a human has to act on. A few rules made the difference between a plan we trusted and a plan we merely hoped worked:
- Priority groups follow the dependency chain, not the org chart. Domain controllers and DNS first, then anything the tier-1 application needs to authenticate or resolve against, then the application tier itself, then everything else. Nobody's political priorities belong in a boot order.
- Every manual step is a documented callout in the plan, not tribal knowledge. SRM lets you insert pre- and post-power-on steps with instructions displayed to whoever runs the plan. The 2 a.m. version of your team does not remember a conversation from March; write the step down.
- IP customization is configured per VM and verified, not assumed inherited. SRM will happily run a recovery plan with incomplete IP mappings; it just leaves those VMs on whatever address they had, which at the recovery site is frequently nothing useful.
Recovery Plan: Financial-Services-Primary-Failover
Priority Group 1 - Infrastructure
DC01-DR, DC02-DR, DNS validated reachable
Priority Group 2 - Data tier
SQL-CLUSTER (AlwaysOn secondary promoted)
Priority Group 3 - Application tier
APP01, APP02 (IP customization applied)
Priority Group 4 - Everything else
FILE01, PRINT01, RPT01
Manual step, Priority Group 1: confirm DNS scavenging disabled at DR site
Manual step, Priority Group 3: validate app connection string against DR SQL listener
That skeleton is most of the value of the engagement, honestly. Everything before it is design work to make the skeleton true.
The annual test that actually fails things over
A recovery plan nobody has run is a diagram with better formatting. SRM's test failover — running the plan against an isolated bubble network, replica disks snapshotted so production replication keeps running underneath — removes the excuse. There is no good reason to have an untested recovery plan in 2017; the tooling to test one without touching production has existed for years.
We test twice a year: a full failover in the bubble network every spring, a scaled tier-1-only test in the fall. The bubble matters more than people expect — an isolated port group with no route back to production, so DNS, AD, and the applications all come up believing they're home, without a packet reaching a real user or a real production host. Skip the bubble and you either risk production or test nothing, because the plan quietly skips every step that would actually touch the network.
The spring test is where the accounting happens: actual time from declaring the disaster to the application being usable, measured with a stopwatch, not estimated from the plan. Our first spring test here ran a tier-1 RTO of 55 minutes against a target of under an hour — close enough to trust, honest enough that we didn't round it up in the report.
The DNS single point of failure we found in testing
Here's the part of the field notes that earns the name. The client's design had two domain controllers at the recovery site, both replicated correctly, both healthy in vSphere Replication, both in Priority Group 1 of the recovery plan. On paper, no single point of failure.
The test failover powered both DR domain controllers up cleanly. Neither served DNS to the bubble network: internal resolution ran through a single forwarder appliance — a physical box never in scope for replication because nobody thought of it as part of Active Directory. It had sat in a rack doing its job for six years, invisible to every DR conversation because it wasn't a domain controller and wasn't on anyone's asset list under a name that suggested importance.
Applications in the bubble came up, tried to resolve names, and sat there. Not failed, exactly — just quiet, the kind of quiet that eats forty minutes before anyone thinks to check DNS instead of the application. We found it because the runbook had a manual verification step — confirm name resolution from a test client inside the bubble — that somebody actually followed instead of assuming.
The fix was unglamorous: DNS Server role added to both DR domain controllers, forwarders mirrored to production, and the appliance itself added to the replication list as a Tier 2 VM, since retiring six years of institutional dependency wasn't in this quarter's budget. The real fix was procedural — every design review for this client now includes an explicit question about what resolves names on a given segment, because that never shows up on a diagram that only draws boxes labeled server.
# The check we now run against every DR site before signing off on a test
Resolve-DnsName -Name app01.corp.example.com -Server 10.20.30.11 -ErrorAction Stop
Resolve-DnsName -Name corp.example.com -Type SOA -Server 10.20.30.11
# If either fails from inside the bubble, stop the test and fix DNS before anything else runs
Nobody's DR plan lists the DNS forwarder nobody thought about as a risk, because by definition nobody thought about it. The lesson we took, and now apply everywhere: a DR test's job is not to confirm the plan works. It's to find the thing the plan didn't know to include.
If you're facing this
If your DR plan is really a backup plan wearing a disaster-recovery label, or you've never run a real test failover and aren't sure what it would find, that's worth finding out on your schedule rather than a real event's. We've built vSphere Replication and Site Recovery Manager programs for government and enterprise clients running distributed estates well past 700 servers, and we can tell you honestly, before you spend a dollar, whether your RPOs are real. Get in touch.