End to End QoS solution for Vmware vSphere with NSX on top of Cisco UCS

1. © 2014 VMware Inc. All rights reserved.
NSX Architecture Design
Lessons Learned from real project
David Pasek
Infrastructure Architect
VCDX #200
End to End QoS Solution for VMware vSphere
with NSX on top of CISCO UCS

2. Agenda
1 Project Overview
2 NSX Conceptual & Logical Design
3 Deep Dive in to Network QoS – Design Decision Point
4 Q & A
CONFIDENTIAL 2

3. • Private Cloud – EMC FEHC-­CA with custom enhancements
• vSphere VM as a Service
• Hyper-­V VM as a Service
• Physical Server as a Service
• Backup as a Service
• Storage as a Service
• Environment / Facilities
• Two datacenters in metro distance (<5 ms)
• Remote Offices (Technical Rooms) in MPLS distance
• Products and Technologies
• CMP: vRealizeAutomation, vRealize Orchestrator, vRealize Business
• Infrastructure Virtualization: VMware vSphere, Hyper-­V, NSX-­v
• Servers: Cisco UCS
• Networking: Cisco Nexus
• Storage: EMC VIPR, EMC VPLEX, EMC VNX, VMware VSAN
• Backup: EMC Avamar, EMC Networker, EMC DataDomain
• Security: NSX + PaloAlto Networks
Project Overview

4. Overall Project High Level Concept
Datacenter A Datacenter B
vSphere Resource Pool - GOLD TIER
VMware vSphere Metro Cluster Stretched across two datacenters
Storage Stretched across two datacenters (VPLEX)
Technical Room
Resource Pool - TR TIER
(vSphere + VSAN)
Remote Location
Existing Core Network
Cloud
Consumers
Cloud
Administrators
vRealize
Automation
vRealize Business Std. + Adv.
IT Finance
vRealize Automation
vCenter Orchestrator
vRealize LogInsight
vRealize Operations Manager
vSphere Resource Pool - SILVER TIER
Cluster in single datacenter
Storage in single datacenter
(different storage tiers)
vSphere Resource Pool - SILVER TIER
Cluster in single datacenter
Storage in single datacenter
(different storage tiers)
Cloud Management Infrastructure Cluster
VMware vSphere Metro Cluster Stretched across two datacenters
Storage Stretched across two datacenters (VPLEX)
Cloud Management Software Stack
Cloud Management Platform, vSphere Management, NSX Management workloads
Hyper-V Resource Pool
Cluster in single datacenter
Storage in single datacenter
Hyper-V Resource Pool
Cluster in single datacenter
Storage in single datacenter
Physical Servers Resource Pool
Server in single datacenter
Storage in single datacenter
Physical Servers Resource Pool
Server in single datacenter
Storage in single datacenter

5. NSX-­v Conceptual Architecture
Datacenter A (CDP-A) Datacenter B (CDP-B)
CORE NETWORK
(dynamic routing protocol has
to be implemented)
PaloAlto FW
Physical Appliance
PaloAlto FW
Physical Appliance
NSX Edge GW NSX Edge GW NSX Edge GW NSX Edge GW
VIRTUALNETWORKOVERLAY
PHYSICAL NETWORK
UNDERLAY
ESXi Host ESXi HostESXi Host ESXi Host
GOLD vSphere Cluster - STRETCHED
ESXi HostESXi Host ESXi Host
SILVER vSphere Cluster
LOCAL
ESXi Host ESXi HostESXi Host ESXi Host
LOGICAL SWITCH (VXLAN SEGMENT)
vNIC
vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW vPaloAlto L7 FW
LOGICAL SWITCH (VXLAN SEGMENT)
NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW NSX FW
NSX DLR
Distributed Logical Router
East-West Routing in DCs
Traffic Steering
NSX DISTRIBUTED LOGICAL FIREWALL
NSX DISTRIBUTED LOGICAL FIREWALL
L3 Fabric
ECMP
+
Dynamic Routing
between PAN,
NSX Edge GWs
and NSX DLRs
PaloAlto
Panorama
Centralized
Security
Management
VMware
NSX Manager
Centralized
Virtual Network
Management
NSX Edge
L2 VPN
NSX Edge
L2 VPN
NSX Edge L2 VPN
Highly Available
TR VPN Termination
NSX Edge
Services GWs
Highly Available
North South Routing
NSX Edge GW
L2 VPN
LOGICAL SWITCH (VXLAN SEGMENT)
Technical Room (TR)
L2 VPN
TUNNEL
(TR<—>DC)
FEHC Management vSphere Cluster - STRETCHED
ESXi Host ESXi Host
SILVER vSphere Cluster - LOCAL
LOGICAL SWITCH (VLAN SEGMENT)
ESXi HostESXi Host ESXi Host
SILVER vSphere Cluster
LOCAL
LOGICAL SWITCH (VXLAN SEGMENT)
NSX DISTRIBUTED LOGICAL FIREWALL

6. NSX-­v Security Concept
vRA Business Group: HR
Logical Network
Micro Security Zone
Technical Service - SAP
[NSX Security Group of all VMs
having tag MSZ-SAP]
Micro Security Zone
Technical Service - A
[NSX Security Group of all VMs
having tag MSZ-A]
vRA Business Group: FINANCE
Logical Network
Micro Security Zone
Technical Service - B
[NSX Security Group of all VMs
having tag MSZ-B]
NSX Distributed
Logical Router
MSZ-SAP MSZ-SAP MSZ-SAP
MSZ-SAP MSZ-SAP
MSZ-A MSZ-A
MSZ-B MSZ-B MSZ-B
Default NSX Security Policy
NAME SOURCE DESTINATION SERVICE ACTION
Default Any Any Any Block
SECURITY TAGS
Security tags for technical services:
MSZ-<Technical-Service-from-CMDB>
For example: MSZ-SAP, MSZ-A, MSZ-B
Security tags for applications:
APP-<gkpke.APP-SEC-TAG[x]>
For example: APP-MSSQL, APP-IIS, APP-EXCHANGE
APP-MSSQL
APP-MSSQL NSX SECURITY GROUPS
We have NSX Security Group for each Technical Service.
This security group forms Micro Security Zone for particular
Technical Service.
For example: MSZ-SAP, MSZ-A, MSZ-B
All VMs tagged with the Security Group name will belong to
this security group.
NSX Security Policy for Micro Security Zones
NAME SOURCE DESTINATION SERVICE ACTION
Inside MSZ-A MSZ-A MSZ-A Any Allow
Inside MSZ-B MSZ-B MSZ-B Any Allow
Inside MSZ-SAP MSZ-SAP MSZ-SAP Any Allow
Other NSX Security Groups and Policies
Other NSX security groups and polices can be created based on
applications tags and other metadata available for NSX.
Physical or Hyper-V Server
belonging in to Micro
Security Zone

7. End to End Network QoS -­ Design Decision Point
• Requirements
• End to end network QoS is required to achieve guarantees for particular network traffics. These
traffics are
• FCoE Storage
• vSphere Management
• vSphere vMotion
• VM production
• VM guest OS agent based backup <== this is the most complex requirement in context of QoS
• Constraints
• CISCO Nexus 7k
• VMware NSX-­v
• CISCO UCS servers B200 M4 with virtual interface card VIC1340 (2x10Gb ports -­ each port
connected to different fabric interconnect)
• Cloud Automation (vRA, vRO)

8. End to End Network QoS – Option 1 of 3
UCS Blade Server
B200 M4
NIC-A1 - 10Gb NIC port
vHBA0
FCoE
CoS 3 40%
Mark as CoS 3
vNIC0
Mgmt
VLAN 100
CoS 1 10%
Mark CoS 1
vNIC2
vMotion
VLAN 101
CoS 2 10%
Mark CoS 2
vNIC4
VM Traffic
VLAN 102
CoS 0 20%
Mark CoS 0
NIC-B1 - 10Gb NIC port
vHBA1
FCoE
CoS 3 40%
Mark as CoS 3
vNIC1
Mgmt
VLAN 100
CoS 1 10%
Mark CoS 1
vNIC3
vMotion
VLAN 101
CoS 2 10%
Mark CoS 2
vNIC5
VM Traffic
VLAN 102
CoS 0 20%
Mark CoS 0
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
vNIC7
Backup
VLAN 103
CoS 4 20%
Mark CoS 4
vNIC6
Backup
VLAN 103
CoS 4 20%
Mark CoS 4
VMwarevSphere-ESXi
vmkernel
Mgmt
(Native VLAN)
vmkernel
vMotion
(Native VLAN)
vmkernel
VTEP
VMware Distributed vSwitch
DVS portgroup (Native VLAN)
VTEP
DVS portgroup (native VLAN)
Backup
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production
VM vNIC
Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 50% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 30% (Backup)
vmnic0 vmnic2 vmnic4 vmnic6 vmnic1 vmnic3 vmnic5 vmnic7
VMware Standard vSwitch VMware Standard vSwitch VMware Distributed vSwitch
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 50% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 30% (Backup)
UCS QoS Policy 1 (vNIC):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None

9. End to End Network QoS – Option 2 of 3
UCS Blade Server
B200 M4
10Gb NIC port (NIC-A1)
vHBA0
FCoE
CoS 3 40%
Mark as CoS 3
10Gb NIC port (NIC-B1)
vHBA1
FCoE
CoS 3 40%
Mark as CoS 3
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
VMwarevSphere-ESXi
vmkernel
Mgmt
vmkernel
vMotion
vmkernel
VTEP
DVS portgroup
VLAN 102, Mark as CoS 0
VTEP
DVS portgroup
VLAN 103, Mark as COS 4
Backup
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production
VM vNIC
Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
vmnic0 vmnic1
VMware Distributed vSwitch (DVS)
DVS portgroup
VLAN 100, Mark as CoS 1
Mgmt
DVS portgroup
VLAN 101, Mark as CoS 2
vMotion
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
DVS per PortGroup Marking
CoS 0: System: VM Traffic
CoS 1: System: Mgmt
CoS 2: System: vMotion
CoS 4: User-def: Backup
vmnic2 vmnic3
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
UCS QoS Policy 1 (vNIC 0,1):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None
vNIC0
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
vNIC1
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%

10. End to End Network QoS – Option 3 of 3
UCS Blade Server
B200 M4
10Gb NIC port (NIC-A1)
vHBA0
CoS 3 40%
FCoE
Mark as CoS 3
10Gb NIC port (NIC-B1)
vHBA1
CoS 3 40%
FCoE
Mark as CoS 3
UCS Fabric Interconnect A (EHM) UCS Fabric Interconnect B (EHM)
vFC vEth vEth vEth vFC vEth vEth vEth
CISCOUCS
CISCO Nexus 7k CISCO Nexus 7k
Eth Eth Eth EthFc Fc
SAN A SAN B
vPC Domain
vPCvPC
VMwarevSphere-ESXi
vmkernel
Mgmt
vmkernel
vMotion
vmkernel
VTEP
DVS portgroup
VLAN 102
VTEP
vEth vEth
VMwareNSX
NSX Logical Switch (VXLAN)
logical segment - Business Group
VM vNIC
Production & Backup
UCS uplink & N7K downlink
QoS Settings
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
vmnic0 vmnic1
VMware Distributed vSwitch (DVS)
DVS portgroup
VLAN 100, Mark as CoS 1
Mgmt
DVS portgroup
VLAN 101, Mark as CoS 2
vMotion
Cisco VIC 1340 (4x10Gb port)
DVS portgroup
Virtual Wire - Business Group 1
if DST IP = Backup Server mark as CoS 4 else CoS 0
DVS per PortGroup Marking
CoS 0: System: VM Traffic
CoS 1: System: Mgmt
CoS 2: System: vMotion
CoS 4: User-def: Backup
vmnic2 vmnic3
CISCO UCS QoS Polices
Bandwidth Management & QoS Marking
UCS QoS Policy UP (Uplinks):
CoS 0: 40% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 4: 40% (Backup)
UCS QoS Policy 1 (vNIC 0,1):
CoS 0: 20% (VM Traffic)
CoS 1: 10% (Mgmt)
CoS 2: 10% (vMotion)
CoS 3: 40% (FCoE)
CoS 4: 20% (Backup)
UCS all vNIC Templates:
Host Control: None
vNIC0
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%
vNIC1
trunk
CoS0 20%
CoS1 10%
CoS2 10%
CoS4 20%

11. End to End Network QoS – Final Decision
• Decision
• Option 3 – QoS (802.1p) marking in VDS and end-­2-­end bandwidth management in UCS
• Justification
• Decision is fully compliant with End to end network QoS requirement
• VXLAN protocol is designed to keep L2 CoS tags by copying inner Ethernet header into outer
Ethernet header => virtual overlay CoS tag is kept even in physical network underlay and it can be
leveraged in Cisco UCS bandwidth management (aka DCB ETS -­ Enhanced Transmission
Selection) to guarantee bandwidth for particular CoS traffics.
• Single vNIC in VM has positive impact on
• NSX Security Policies
• Simple In-­guest OS routing (default gateway only) without need for additional static routes
• vRealize Automation Custom Integrations are simpler (single hostname, simpler integration
with IPAM, etc.)
• Impact
• DVS QoS Policy (conditional 802.1p marking) has to be configured manually for each DVS
portgroup used as NSX virtual wire (aka VXLAN) – can be automated by custom integration
(SOLUTION IMPROVEMENT)
• Detail Test Plan has to be prepared to validate correct QoS behavior (RISK MITIGATION)

12. Questions and Answers
Blog post with additional details:
http://blog.igics.com/2015/12/end-­to-­end-­qos-­solution-­for-­vmware.html
Twitter: @david_pasek
Blog: http://blog.igics.com