For the past year we have been designing and running a multi-tenant Skype for Business Server 2015 platform for a service provider that hosts communications for a portfolio of mid-sized organizations — the kind of customers who want enterprise voice, IM, and conferencing but will never staff a UC engineer. The platform currently carries north of 10,000 concurrent users across tenants, alongside a VDI estate on Citrix XenDesktop, and it is held to a 99.95% availability target with real financial teeth in the contract.
Skype for Business 2015 architecture is well documented for the single-enterprise case. The multi-tenant case is documented barely at all, and most of what exists assumes you are a carrier with a carrier's budget. These are the design decisions that mattered, the load testing that told us the truth, and the two incidents that taught us more than the design phase did.
A scope note: this is a hosted dedicated-infrastructure model — shared platform, logically isolated tenants — not the old Lync multitenant hosting pack, which trades away too much functionality (no on-premises integration story, constrained federation) for the isolation it provides. We isolate with topology and policy instead, and we accept the operational discipline that requires.
Tenant isolation starts with SIP domains and ends with policy
Every tenant gets its own SIP domain, its own user OU, and its own complete set of scoped policies. That last part is the discipline: in Skype for Business, almost everything dangerous is a global policy by default — external access, conferencing capability, client version control, dial plans. On a single-enterprise deployment, editing the global policy is Tuesday. On a multi-tenant platform, editing the global policy is a change that touches every customer simultaneously, which is another way of saying it is an outage waiting for a typo.
Our standing rule: the global policy of every type is configured to the most restrictive sensible state, and no user is ever served by it. Every tenant gets explicitly assigned per-user policies, created from a template set with New-CsExternalAccessPolicy, New-CsConferencingPolicy, New-CsClientPolicy and friends, named with a strict Tenant-PolicyType-Variant convention. Onboarding a tenant is a scripted operation — SIP domain added to the topology, DNS records verified, policy set stamped out, test users validated — because anything done by hand twice will be done wrong the third time.
Address book segregation deserves a mention because every prospective tenant asks about it. Users in tenant A should not autocomplete users in tenant B. We handle this with per-tenant address book segmentation via ABS filtering and by keeping tenants in separate OUs with search scopes applied at the client policy level. It is not cryptographic isolation and we say so plainly in the service description; a tenant who needs hard isolation needs their own deployment, and we have told prospects exactly that.
Front-end pool design: respect the fabric
The core of the platform is two paired Front End pools, each with six Front End servers, running Enterprise Edition against SQL Server 2014 AlwaysOn back ends. Six is not an arbitrary number. Skype for Business 2015 pools run on Windows Fabric, and the pool's brain — the routing groups that own each user's registration and presence — is distributed across the Front Ends with quorum rules that are unforgiving if you sized for a Lync 2010 mental model.
The rules we build to, learned partly from documentation and partly from a bad afternoon:
- A pool needs a majority of Front Ends running for the pool to be functional. Six servers means we can lose two and keep quorum with margin.
- Never patch more than one Front End at a time, and always wait for
Get-CsPoolFabricStateto come back clean before touching the next. The bad afternoon: an eager administrator (ours, we own it) rebooted a second server while routing groups from the first were still re-homing. The pool did not go down, but several hundred users across two tenants dropped to resiliency mode — signed in, no presence, no conferencing — for twenty minutes. The postmortem produced a patching runbook with mandatory fabric-state gates, and the runbook has held since. Reset-CsPoolRegistrarStateis a scalpel, not a hammer. Know the difference between ServiceReset, QuorumLossRecovery, and FullReset before the incident, because reading TechNet during one is how a twenty-minute problem becomes a four-hour one.
The two pools are paired for disaster recovery with backup service replication, and we actually exercise Invoke-CsPoolFailOver twice a year against production, tenant-communicated, because a DR capability you have never invoked is a hypothesis, not a capability.
Edge topology: where multi-tenant gets expensive
Every tenant SIP domain needs sip.tenantdomain.example and webconf / av edge answering with a certificate that covers it. That means the Edge pool's public certificate is a SAN certificate that grows with every tenant onboarded — and re-issuing and re-binding a public SAN cert across an Edge pool is now a routine, scripted, maintenance-window operation for us rather than an annual scramble. We batch tenant onboarding partly for this reason.
The Edge pool itself is three servers, DNS load balanced for SIP (hardware load balancing buys little for SIP and complicates federation) with a hardware load balancer only where it is actually required, on the reverse proxy path for web services. Federation is enabled per-tenant by policy, closed by default. The A/V edge sizing is driven by conferencing patterns, not user counts: our telemetry says the platform peak is media relay for multiparty conferences between 10 and 11 a.m., and that hour, not the concurrent-user number, is what the Edge NICs are sized against.
Load testing to 10,000: the Stress and Performance Tool doesn't lie
Before the platform took its first production tenant, we spent three weeks with the Skype for Business Server Stress and Performance Tool driving synthetic load from a bank of client simulation machines: registration storms, IM and presence churn, conference join waves, PSTN-scale call setup rates.
Findings that changed the design:
| What we tested | What we expected | What we found |
|---|---|---|
| 10,000 user registration storm (cold start) | CPU-bound Front Ends | SQL back end disk latency was the ceiling |
| Presence churn at business-hours rates | Negligible | Negligible — presence is cheap until federation multiplies it |
| Conference join wave (Monday 9 a.m. pattern) | Focus/MCU CPU pressure | Correct, plus certificate operations spiking on joins |
| Failover mid-load | Brief resiliency-mode blip | Confirmed, ~90 seconds for routing group re-home |
The registration storm result mattered most. A datacenter power event that reboots everything produces exactly that cold-start pattern, and our original SQL storage tier would have stretched full re-registration of the estate to nearly an hour. We moved the back-end databases to a faster tier and re-ran the test until cold start completed in under fifteen minutes. That is the availability target being defended in the design phase — the 99.95% is lost or kept in the worst hour of the year, not the average one.
Watching media quality before the tenants do
Availability says whether the service is up. It says nothing about whether calls sound terrible. The Monitoring databases — CDR and QoE — are the most underused feature in most deployments we audit, and on a multi-tenant platform they are the difference between "tenant B's users complain about robot-voice on Thursdays" being a mystery or a query.
We run scheduled reporting from the QoE database per tenant: poor-call percentage (using the standard network mean opinion score degradation classifiers), top subnets by poor calls, and wired-versus-wireless breakdown. Twice now that reporting has let us tell a tenant, with numbers, that their call quality problem lived in their own office Wi-Fi and not on our platform — a conversation that goes very differently when you bring per-subnet packet loss data instead of assurances. We also alert on our own side: relay traversal failures at the Edge and any sustained rise in avg jitter on the A/V MCUs page the on-call before humans notice.
The Key Health Indicators pack in SCOM covers the platform layer, and we tuned it the way we tune all of SCOM — aggressively, until every alert that fires is one someone acts on. But KHI tells you the servers are healthy; QoE tells you the users are. You need both, and platforms of this size are typically monitored with the first and blind on the second.
If you're facing this
Whether you are a provider building shared UC infrastructure or an enterprise whose Skype for Business 2015 deployment needs to stop failing its availability number, the design levers are the same: pool quorum discipline, policy hygiene, honest load testing, and monitoring that measures what users experience. We have designed, load tested, and operated this at 10,000-plus concurrent users and 99.95% — get in touch and we will tell you what it took, including the parts that went wrong.