This presentation discusses networking design and configuration considerations for VMware vSAN. It provides an overview of vSAN networking components and traffic, requirements for ports and firewalls, and considerations for multicast, unicast, NIC teaming and load balancing. It also reviews supported network topologies like single site, stretched and 2-node clusters and discusses performance considerations.

1. Cormac Hogan
Andreas Scherr
STO1193BU
#STO1193BU
A Closer Look at vSAN
Networking Design and
Configuration
Considerations

2. • This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these
features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or
sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not
been determined.
Disclaimer
2

3. Agenda
1 vSAN Networking Overview
2 Multicast and Unicast
3 NIC Teaming and Load Balancing
4 Network Topologies (incl. Stretched and 2-node)
5 Network Performance Considerations
3

4. Where should I begin? StorageHub!
• https://storagehub.vmware.com/#!/vmware-vsan/plan-and-design
4

5. vSAN Networking Overview
5

6. vSAN Networking – Major Software Components
• CMMDS (Cluster Monitoring, Membership, and Directory Service)
• Inter cluster communications and metadata exchange
– Multicast with <= vSAN 6.5
– Unicast with >= vSAN 6.6
– Heartbeat sent from master to all hosts every second
• Traffic light in steady state
• RDT (Reliable Datagram Transport)
• Bulk of vSAN traffic
– Virtual Disk data distributed across cluster
– Replication /Resynch Traffic
6

7. vSAN Networking – Ports and Firewalls
• ESXi Firewall considerations
– On enablement of vSAN on a given cluster, all required ports are
enabled/disabled automatically; no admin action
• Ports
– CMMDS (UDP 12345, 23451, 12321)
– RDT (TCP 2233)
– VSANVP (TCP 8080)
– Witness Host (TCP port 2233 and UDP Port 12321)
– vSAN Encryption / KMS Server
• Communication between vCenter and KMS to obtain keys
• vSAN Encryption has special dynamic firewall rule opened on
demand on ESXi hosts
7

8. Network Connectivity – IPv6
• vSAN can operate in IPv6-only mode
– Available since vSAN 6.2
– All network communications are through IPv6 network
• vSAN supports mixed IPv4 & IPv6 during upgrade only
– Do not run mixed mode in production
8

9. Minimum NIC requirements for vSAN Networking
9
+10Gb
support
1Gb
support Comments
Hybrid Cluster Y Y
10Gb min. recommended, but 1Gb supported,
<1ms RTT
All-Flash Cluster Y N
All Flash requires 10Gb min. 1Gb not supported,
<1ms RTT
Stretched Cluster - Data to Data Y N
10Gb required between data sites*,
<5ms RTT
Stretched Cluster - Witness to Data Y Y
100Mbps connectivity required from data sites to witness.
<200ms RTT
2-node Data to Data Y Y
10Gb min. required for All-Flash. 1Gb supported for hybrid,
but 10Gb recommended
2-node Witness to Data Y Y
1.5Mbps bandwidth required.
<500ms RTT

10. Distributed or Standard Switches?
10
• vSphere Standard Switch
• No management dependence on vCenter
• Recovery is simple
• Prone to misconfiguration in larger setups
• vSphere Distributed Switch
• Consistency
Avoids configuration skew
• Teaming and Failover
LACP/LAG/ether-channel
• Network I/O Control
Manage/allocate network bandwidth for
different vSphere traffic types
vSphere Distributed Switch is Free with vSAN

11. Network I/O Control (NIOC) Configuration Sample
• Single 10-GbE physical adapters for simplicity
• NICs handles traffic for vSAN, vMotion, and virtual machines and management traffic
• If adapter becomes saturated, Network I/O Control controls bandwidth allocation
• Sample configuration:
11
Traffic Type Custom Shares Value Bandwidth
vSAN 100 5Gbps
vMotion 50 2.5Gbps
Virtual Machine 30 1.5Gbp
Management 20 1Gbps

12. NIC Teaming and Failover options
12
• Keep it simple folks!
• All Virtual Switches Support (vSS + vDS)
– Routed based on IP Hash / Virtual Port ID
• Distributed Switch Only (vDS)
– Route based on Physical NIC Load (LBT)
• Distributed Switch + Physical Switch Only
– Physical switches that support LACP/LAG/ether-
channel provide additional load balancing algorithms
Multi chassis link aggregation capable switches

13. vSAN Multicast & Unicast
13

14. What is Multicast?
14
• vSAN 6.5 (and earlier) used multicast traffic as a discovery
protocol to find all other nodes trying to join a vSAN cluster.
• Multicast is a network communication technique utilized to
send information simultaneously (one-to-many or many-to-
many) to a group of destinations over an IP network.
• Multicast needs to be enabled on the switch/routers of the
physical network.
• Internet Group Management Protocol (IGMP) used within
an L2 domain for group membership (follow switch vendor
recommendations)
• Protocol Independent Multicast (PIM) used for routing
multicast traffic to a different L3 domain
Multicast added complexity to vSAN networking

15. IGMP Considerations
• Consideration with multiple vSAN clusters
– Prevent individual clusters from receiving all multicast streams
– Option 1 – Separate VLANs for each vSAN cluster
– Option 2 - When multiple vSAN clusters reside on the same layer 2 network, VMware
recommends changing the default multicast address
• See VMware KB 2075451
15

16. Multicast Group Address on vSAN
• The vSAN Master Group Multicast Address created is 224.1.2.3 – CMMDS updates.
• The vSAN Agent Group Multicast Address is 224.2.3.4 – heartbeats.
• The vSAN traffic service will assign the default multicast address settings to each host node.
16
# esxcli vsan network list
Interface
VmkNic Name: vmk2
IP Protocol: IP
Interface UUID: 26ce8f58-7e8b-062e-ba57-a0369f56deac
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5

17. vSAN 6.6 introduces Unicast
in place of Multicast for vSAN
communication
17

18. vSAN and Unicast
• vSAN 6.6 now communicates using unicast for
CMMDS updates.
• A unicast transmission/stream sends IP packets to a
single recipient on a network.
• vCenter becomes the new source of truth for vSAN
membership.
– List of nodes is pushed to the CMMDS layer
• The Networking Mode (unicast/multicast) is not
configurable
18
vSAN 6.6 and above
Unicast

19. vSAN and Unicast
• The Cluster summary now shows if a vSAN cluster network mode is Unicast or Multicast:
19

20. Member Coordination with Unicast on vSAN 6.6
• Who tracks cluster membership if we no
longer have multicast?
• vCenter now becomes the source of truth for
vSAN cluster membership with unicast
• The vSAN cluster continues to operate in
multicast mode until all participating nodes are
upgraded to vSAN 6.6
• All hosts maintain a configuration generation
number in case vCenter has an outage.
– On recovery, vCenter checks the configuration
generation number to see if the cluster
configuration has changed in its absence.
20
vCenter

21. New Unicast considerations
in vSAN 6.6
21

22. Upgrade / Mixed Cluster Considerations with unicast
22
vSAN Cluster
Software
Configuration
Disk Format
Version(s)
CMMDS
Mode
Comments
6.6 Only Nodes* All Version 5 Unicast
Permanently operates in unicast. Cannot switch to
multicast. Adding older nodes will partition cluster.
6.6 Only Nodes*
All Version 3 or
below
Unicast
6.6 nodes operate in unicast mode.
Switches back to multicast if < vSAN 6.6 node added.
Mixed 6.6 and vSAN
pre-6.6 Nodes
Mixed Version 5 with
Version 3 or below
Unicast
6.6 nodes with v5 disks operate in unicast mode. Pre-6.6
nodes with v3 disks will operate in multicast mode.
*** This causes a cluster partition! ***
Mixed 6.6 and vSAN
pre-6.6 Nodes
All Version 3 or
Below
Multicast
Cluster operates in multicast mode. All vSAN nodes must
be upgraded to 6.6 to switch to unicast mode.
*** Disk format v5 will make unicast mode permanent ***

23. vSAN 6.6 only nodes – additional considerations with unicast
• All hosts running vSAN 6.6, cluster will communicate using unicast
– Even if disk groups are formatted with < version 5.0, e.g. version 3.0
• vSAN will revert to multicast mode if a non-vSAN 6.6 node is added to the 6.6 cluster
– But only if no disk group format == version 5.0
• A vSAN 6.6+ cluster will only ever communicate in unicast if a version 5.0 disk group exists
• If a non-vSAN 6.6 node is added to a 6.6 cluster which contains at least one version 5.0 disk
group, this node will be partitioned and will not join the vSAN cluster
23

24. Considerations with Unicast
• Considerations with vSAN 6.6 unicast and DHCP
– vCenter Server deployed on a vSAN 6.6 cluster
– vSAN 6.6 nodes obtained IP addresses via DHCP
– If IP addresses change, vCenter VM may become unavailable
• Can lead to cluster partition as vCenter cannot update membership
– This is not supported unless DHCP reservations are used.
• Considerations with vSAN 6.6 unicast and IPv6
– IPv6 is supported with unicast communications in vSAN 6.6.
– However IPv6 Link Local Addresses are not supported for
unicast communications on vSAN 6.6
• vSAN doesn’t use link local addresses to track membership
24
vCenter

25. Query Unicast with esxcli
• vSAN cluster node now displays the CMMDS networking mode - unicast or multicast.
– esxcli vsan cluster get
25

26. Query Unicast with esxcli
• One can also check which vSAN cluster nodes are operating in unicast mode
– esxcli vsan cluster unicastagent list:
• Unicast info is also displayed in vSAN network details
– esxcli vsan network list
26

27. NIC Teaming and Load-Balancing
Recommendations
27

28. NIC Teaming – single vmknic, multiple vmnics (uplinks)
• Route based on originating virtual port
– Pros
• Simplest teaming mode, with very minimum physical
switch configuration.
– Cons
• A single VMkernel interface cannot use more than a single
physical NIC's bandwidth.
• Route Based on Physical NIC Load
– Pros
• No physical switch configuration required.
– Cons
• Since only one VMkernel port, effectiveness of using this is
limited.
• Minor overhead when ESXi re-evaluates the load
28

29. Load Balancing - single vmknic, multiple vmnics (uplinks)
• vSAN does not use NIC teaming for load
balancing
• vSAN has no load balancing mechanism
to differentiate between multiple vmknics.
• As such, the vSAN IO path chosen is not
deterministic across physical NICs
29
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1000000
Node 1 Node 2 Node 3 Node 4
KBps Utilization per vmnic -Multiple VMknics
vmnic0 vmnic1

30. NIC Teaming – LACP & LAG (***Preferred***)
• Pros
– Improves performance and bandwidth
– If a NIC fails and the link-state goes down, the
remaining NICs in the team continue to pass traffic.
– Many load balancing options
– Rebalancing of traffic after failures is automatic
– Based on 802.3ad standards.
• Cons
– Requires that physical switch ports be configured in
a port-channel configuration.
– Complexity on configuration and maintenance
30

31. Load Balancing – LACP & LAG (***Preferred***)
• More consistency compared to “Route
based on physical NIC load”
• More individual Clients (VMs) will cause
further increase probability of a balanced
load
31
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
Node 1 Node 2 Node 3 Node 4
KBps Utilization per vmnic - LACP Setup
vmnic0 vmnic1

32. vSAN network on different subnets
• vSAN networks on 2 different subnets?
– If subnets are routed, and one host’s NIC fails,
host will communicate on other subnet
– If subnets are air-gapped, and one host’s NIC
fails, it will not be able to communicate to the
other hosts via other subnet
– That host with failing NIC will become isolated
– TCP timeout 90sec on failure
32

33. Supported Network Topologies
33

34. Topologies
• Single site, multiple hosts
• Single site, multiple hosts with Fault Domains
• Multiple sites, multiple hosts with Fault Domains (campus cluster but not stretched cluster)
• Stretched Cluster
• ROBO/2-node
• Design considerations
– L2/L3
– Multicast/Unicast
– RTT (round-trip-time)
34

35. Simplest topology - Layer-2, Single Site, Single Rack
• Single site, multiple hosts, shared subnet/VLAN/L2 topology, multicast with IGMP
• No need to worry about routing the multicast traffic in pre-vSAN 6.6 deployments
• Layer-2 implementations are simplified even further with vSAN 6.6, and unicast. With such a
deployment, IGMP snooping is not required.
35

36. Layer-2, Single Site, Multiple Racks – pre-vSAN 6.6 (multicast)
• pre-vSAN 6.6 where vSAN traffic is multicast
• Vendor specific multicast configuration required (IGMP/PIM)
36

37. Layer-2, Single Site, Multiple Racks – 6.6 and later (unicast)
• vSAN 6.6 where vSAN traffic is unicast
• No need to configure IGMP/PIM on the switches
37

38. Stretch Cluster Topologies
38

39. Stretched Cluster – L2 for data, L3 to witness or L3 everywhere
• vSAN 6.5 and earlier, traffic between data sites is multicast (meta) and unicast (IO).
• vSAN 6.6 and later, all traffic is unicast.
• In all versions of vSAN, the witness traffic between a data site and the witness site has always
been unicast.
39

40. Stretched Cluster - Why not L2 everywhere? (unsupported)
• Consider a situation where the link between Data Site 1 and Data Site 2 is broken
• Spanning Tree may discover a path between Data Site 1 and Data Site 2 exists via switch S1
• Possible performance decrease if data network traffic passes through a lower specification
witness site
40

41. 2-Node (ROBO)
41

42. 2-Node vSAN for Remote Locations
• Both hosts in remote office store data
• Witness in central office or 3rd site
stores witness data
• Unicast connectivity to witness
appliance
– 500ms RTT Latency
– 1.5Mbps bandwidth from Data Site to
WitnessCluster
vSphere vSAN
vSphere vSAN
vSphere vSAN
Witness
vSphere vSAN
Witness
500ms RTT latency
1.5Mbps bandwidth
500ms RTT latency
1.5Mbps bandwidth
42

43. 2-node Direct Connect and Witness traffic separation
43
VSAN
Datastore
witness
10GbE
vSAN traffic via Direct Cable
management & witness traffic
• Separating the vSAN data traffic from witness traffic
• Ability to connect Data nodes directly using Ethernet cables
• Two cables between hosts for higher availability of network
• Witness traffic uses management network
Note: Witness Traffic Separation is NOT supported for
Stretch Cluster at this time

44. vSAN and Performance
Network relevance
44

45. General Concept on Network Performance
• Understanding vSAN concepts and features
– Standard vSAN setup vs. Stretch Cluster, FTT=1 or RAID5/6
• Understand Network Best Practice for optimum Performance – physical switch topology
– ISL trunks are not over subscripted
– MTU size factor
– No errors/drops/pause frames on the Network switches
45

46. General Concept on Network Performance
• Understand Host communication
– No errors/drops/CRC/pause frames on the Network card
– Driver/Firmware as per our HCL
– Use SFP/Gbic certified by your Hardware Vendor
– Use of NIOC to optimize traffic on the protocol layer if links sharing traffic (Ex. VM/vMotion/..)
46

47. DEMO: Adding 10ms network latency
47

48. Summary: Graphical interpretation IOPS vs. latency
48
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
0 5 10 15 20 25
IOPS
additional latency increase ms
latency ms Linear (latency ms)
+10ms latency = ~23100 IOPS
+5ms latency = ~33000 IOPS
Native = ~47000 IOPS

49. DEMO: Network 2% and 10% packet loss
49

50. Summary: Graphical interpretation IOPS vs. loss %
50
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
50000
0 5 10 15 20 25
IOPS
% loss
loss % Expon. (loss %)
1% loss = ~42300 IOPS
Native = ~47000 IOPS
2% loss = ~32000 IOPS
10% loss = ~3400 IOPS

51. Nerd Out With These Key vSAN Activities at VMworld
#HitRefresh on your current data center and discover the possibilities!
Earn VMware digital badges to
showcase your skills
• New 2017 vSAN Specialist
Badge
• Education & Certification Lounge:
VM Village
• Certification Exam Center:
Jasmine EFG, Level 3
Become a
vSAN Specialist
Learn from self-paced and expert
led hands on labs
• vSAN Getting Started Workshop
(Expert led)
• VxRail Getting Started (Self
paced)
• Self-Paced lab available online
24x7
Practice with
Hands-on-Labs
Discover how to assess if your IT
is a good fit for HCI
• Four Seasons Willow Room/2nd
floor
• Open from 11am – 5pm Sun,
Mon, and Tue
• Learn more at Assessing &
Sizing in STO1500BU
Visit SDDC
Assessment Lounge

52. 3 Easy Ways to Learn More about vSAN
52
• Live at VMworld
• Practical learning of
vSAN, VxRail and more
• 24x7 availability online
– for free!
vSAN Sizer
vSAN Assessment
New vSAN Tools
• StorageHub.vmware.com
• Reference architectures,
off-line demos and more
• Easy search function
• And More!
Storage Hub Technical Library Hands-On Lab
Test drive vSAN
for free today!

54. Cormac Hogan
@CormacJHogan
Andreas Scherr
@vsantester