Mellanox provides networking solutions for cloud infrastructure and data centers. They offer switches powered by their own silicon that support open standards. Mellanox partners with other companies to offer open network operating systems like SONiC that run on their switches. Mellanox contributes code and support to make open networking solutions like SONiC successful in production deployments. Their solutions aim to provide high performance, low latency, and predictable networking without proprietary lock-in.

1. © 2019 Mellanox Technologies | Confidential 1
Avi Alkobi Director Switch EMEA
OpenInfra Poland 2019
May 2019
OPEN CLOUD NETWORKING AT FULL SPEED

2. © 2019 Mellanox Technologies | Confidential 2
Company
Headquarters
Mellanox
Overview
Ticker: MLNX
Yokneam, Israel
Sunnyvale, California
Worldwide Offices
~2,950Employees worldwide

3. © 2019 Mellanox Technologies | Confidential 3
Industry Leaders Choose
Mellanox Solutions
10of Top 10 Automotive Manufacturers
3of Top 5 Pharmaceutical Companies
9of the Top 10 Oil and Gas Companies
9of Top 10 Hyperscale Companies
5of Top 6 Global Banks

4. © 2019 Mellanox Technologies | Confidential 4
Mellanox Solution
Cloud Storage
Big Data
Machine
Learning
DefenseFinancialMultimedia

5. © 2019 Mellanox Technologies | Confidential 5
Leaf/Spine Deployments
Layer 2 / MLAG Layer 3 / ECMP L2 over Layer 3 VXLAN
Connect out via spines, L3 GW on spines or above
Anycast L3 GW on TORs or FW as GW located on the
border leaf.
EVPEN Type 5 for Routes out of the fabric.
BGP from the Host – Kuberentes
VXLAN from the host- VMware , OpenStack, Kubernetes

6. © 2019 Mellanox Technologies | Confidential 6
Mellanox – Not Like Other Network Vendors
▪ Mellanox is Unique: ▪ Cisco, Arista, Juniper, etc.:
• Proprietary OS
• Generic switch ASICs
• Re-label & lock generic optics
• License Features
Unique Switch Silicon
Open Ethernet
End to End

7. © 2019 Mellanox Technologies | Confidential 7
End-to-End Interconnect Solutions
Storage
Front /
Back-End
Server /
Compute
Switch /
Gateway

8. © 2019 Mellanox Technologies | Confidential 8
Mellanox Adapters Unique Network Intelligence
ConnectX-5
ConnectX-4 / Lx
ConnectX-6

9. © 2019 Mellanox Technologies | Confidential 9
The VxLAN overlay challenge
▪ Common offload techniques such as hardware segmentation/re-assembly, checksum offload, and
CPU core scaling (RSS/TSS) do not operate on the VM TCP/IP packets (inner payload)
▪ Leading to significant CPU overhead and much lower performance
▪ Solution: Overlay aware Network Interface Cards
• Overlay Network Accelerators
• Penalty free overlays at bare-metal speed
Generated by the VM
Generated by the
Hypervisor
VXLAN Packet Format

10. © 2019 Mellanox Technologies | Confidential 10
VXLAN Offload on ConnectX-4 Lx Validated through Nuage VSP
▪ 2-5x higher throughput
▪ Line rate throughput at 40Gb/s
▪ Latency reduction
▪ 30-60% improvement in CPU
http://www.nuagenetworks.net/wp-
content/uploads/2016/10/PR1609022803EN_NN_Mellanox_PartnerSolutionBrief.pdf

11. © 2019 Mellanox Technologies | Confidential 11
Open Ethernet SN2000 Series
SN2700 – 32x100GbE (up to 64 x 50/25/10GbE)
The Ideal 100GbE ToR / Aggregation
SN2410 – 8x100GbE + 48x25GbE
25GbE ➔ 100GbE ToR
SN2100 – 16x100GbE ports (64x25GbE)
Ideal storage/Database Switch
Highest 25GbE Density per rack unit
SN2010 – 18x10/25GbE + 4x40/100GbE
Ideal HCI ToR Switch
300nsSN2700 – 169W
SN2410 – 165W
SN2100 – 94W
▪ Predictable Performance
▪ Fair Traffic Distribution for Cloud
▪ Best-in-Class Throughput, Latency, Power Consumption
▪ Zero Packet Loss Energy efficiency

12. © 2019 Mellanox Technologies | Confidential 12
Cloud Without Compromise: Spectrum-2
▪ Fully shared 42MB buffer
▪ Advanced queue management and congestion control
Best in Class Buffers
▪ Unprecedented scale – 512K Shared table
▪ Shared Table for IP/MAC/ACL/NAT/Multicast/Tunnels
Deep Tables
▪ Advanced Network Virtualization
▪ Programmability without performance penalties
Best in Class Virtualization
▪ Streaming Telemetry
▪ Histograms
▪ Inband Network Telemetry
▪ Per Flow Accounting
Best in Class Telemetry

13. © 2019 Mellanox Technologies | Confidential 13
Spectrum 2 - Open Ethernet SN3000 Series
SN3700C – 32x100GbE (128x 1-25GbE)
100GbE Spine/ToR
SN3700 – 32x200GbE (128x 1-50GbE)
200GbE Spine
(Q4 2019)
SN3800 – 64x100GbE
Spine/Super Spine
(Q4 2019)
SN3510 – 48x25/50GbE + 8x400GbE
25/50GbE ➔ 400GbE ToR
(Q1 2020)
▪ Best-in-Class Buffers
▪ Best-In-Class Virtualization
▪ Best-In-Class Telemetry

14. © 2019 Mellanox Technologies | Confidential 14
We Are Building an Open Ecosystem
Software
Hardware
Switch Kernel Drivers Platform Drivers
Switch SDK
SAI
OS + Stack
Stack
OS

15. © 2019 Mellanox Technologies | Confidential 15
Switch NOS Reference Architecture
▪ Protocol Stack
• Network protocols (RIB)
- Bridge, STP, OSPF, BGP.
• Forming FIB our of RIB
▪ Forwarding Agent
• Middleware between Protocol Stack & ASIC
• Programming FIB into the ASIC (HW offload)
• Uses special API to communicate with ASIC
▪ API
• Proprietary ASIC vendor SDK
• Broadcom, Mellanox, Cavium, Marvell, etc.
• Trend to standardize and open SDK
• SAI, OpenNSL, OF-DPA, P4, etc. ASIC
Vendor SDK
Abstraction API
Forwarding Agent
Standard Linux
Linux kernel
Linux Native APIs
(Neltink)
Protocol Stack
Linux Network Stack

16. © 2019 Mellanox Technologies | Confidential 16
Switch NOS architecture examples
OS Kernel
Switch Hardware
CPU Platform HW ASIC
ASIC kernel
drivers
Forwarding
Agent
ASIC SDK
Protocol Stack
Traditional NOS
Linux Kernel
Switch Hardware
CPU Platform HW ASIC
ASIC kernel
drivers
Forwarding
Agent
ASIC SDK
Protocol Stack
Commercial NOS
based on Linux
Linux Kernel
Switch Hardware
CPU Platform HW ASIC
ASIC kernel drivers
Forwarding
Agent ASIC SDK
Protocol Stack
Abstraction API
Open NOS
based on Linux
Binary blobProprietary SWOpen/Free SW

17. © 2019 Mellanox Technologies | Confidential 17
MLNX-OS traditional Industry like OS
▪ CLI
▪ GUI
▪ SNMP

18. © 2019 Mellanox Technologies | Confidential 18
Cumulus – Mellanox Partnership
Economical scalability
With commodity hardware and a
standardized Linux stack, achieving
a lower TCO by up to 60%
1 2 3 4Built for the automation age Standardized toolsets Choice and flexibility
Making networking repeatable
and consistent
Easily enable Linux tools:
automation, monitoring, analytics…
50+ hardware platforms, from
11 vendors, and 2 silicon
Best Hardware

19. © 2019 Mellanox Technologies | Confidential 19
SONiC - Software for Open Networking in the Cloud
▪ SONiC is a collection of software packages installed on Linux running on a network hardware switch
which make it a complete, functional router targeted at data center networks. Runs on Debian 8 ‘Jessie‘
distribution.
▪ SONiC is supported by the community and all code is shared in public github
https://github.com/Azure/SONiC
https://github.com/Azure/SONiC/wiki/Architecture
▪ SONiC deployment
▪ in Microsoft production datacenters today and in Mellanox IT
▪ Alibaba is planning soon as well
▪ EMEA

20. © 2019 Mellanox Technologies | Confidential 20
SONiC Architecture cont.
▪ The Switch State Service (SwSS) is a collection of software
that provides a database interface for communication with
and state representation of network applications and network
switch hardware.
Linux Kernel
Switch Hardware
CPU Platform HW ASIC
ASIC kernel drivers
Forwarding
Agent SAI
Protocol Stack
Abstraction API

21. © 2019 Mellanox Technologies | Confidential 21
Mellanox is the ONLY vendor to contribute at all levels
SONiC participants (From Azure blog)
Source: Azure Blog March 2017

22. © 2019 Mellanox Technologies | Confidential 22
SONiC content
▪ 2017 releases content
The following were released
SONiC.201705 05/15/2017 SAI 0.94
SONiC.201708 09/15/2017 SAI 0.94
SONiC.201712 12/20/2017 SAI 1.03
▪ BGP
▪ ECMP
▪ LAG
▪ LLDP
▪ QoS ECN
▪ QoS RDMA
▪ Priority flow control
▪ WRED
▪ COS
▪ SNMP
▪ Syslog
▪ Sysdump
▪ NTP
▪ COPP
▪ DHCP Relay Agent
▪ SONiC to SONiC upgrade
▪ Multiple Images Support
▪ One Image
---------------------------------------
▪ VLAN
▪ Ingress ACL permit/deny
▪ IPv6
▪ Tunnel Decap
▪ Port Speed Setting
▪ BGP Graceful restart helper
▪ BGP MP
---------------------------------------
▪ Fast Reload
▪ ConfigDB Framework
▪ TACAS+
▪ LACP Fallback
▪ MTU Setting
▪ vLAN Trunk
▪ Dynamic ACL Configuration
▪ SWSS Unit Testing Framework

23. © 2019 Mellanox Technologies | Confidential 23
HW offload without SDK = Linux Switch
Linux Kernel
Switch Hardware
CPU Platform HW ASIC
ASIC kernel drivers
Forwarding
Agents Proprietary
ASIC SDK
Protocol Stack
Abstraction APIs
Open NOS based on Linux
Linux Kernel
Switch Hardware
CPU Platform HW ASIC
Open
ASIC kernel driver
Protocol Stack
Linux Network Stack
Netlink
Open Linux as a NOS
Open source

24. © 2019 Mellanox Technologies | Confidential 24
Linux Routing Applications
Linux Switch architecture
Linux Kernel w/ Switchdev Driver
Linux Operating System
User Space
Hardware
Kernel Space
Linux Management Applications

25. © 2019 Mellanox Technologies | Confidential 25
Available today [Linux Kernel 5.0] – feature list
▪ Visibility and
Maintainability
• [ER]SPAN
• Temperature
• Fans
• LED Control
• ethtool (port counter, FW
version, transceiver data)
• Resource queries
• RIF counters
▪ Protocols
• Bridge - 802.1D​
• VLAN - 802.1Q​
• LAG​
• LLDP​
• IGMP snooping​
• Unicast IPv4/IPv6 router​
• ECMP​
• DCB
• QoS
• IGMP flood control
• sFlow
• 256 VRFs
• GRE tunnelling
• Multicast IPv4/IPv6 router
• IPv4/IPv6 weighted ECMP
• VRRP
• VxLAN
▪ ACL
• tc-flower offload
• Actions: Drop, Forward,
Counters, Trap, TC_ACT_OK
• TC chain template
• Keys: Port, DMAC,
SMAC, Ethertype, IP proto, SIP
DIP (IPv4/6), TCP/UDP, L4
port, VLAN-ID, PCP, DCSP,
VLAN valid, TCP flags​
▪ Misc
• ‘devlink’ tool
• Port splitter
• Shared buffer configuration
• Internal secured FW upgrade
• ECN: RED and PRIO

26. © 2019 Mellanox Technologies | Confidential 26
Open NOS in production?
▪Why not?
▪Microsoft, Alibaba - SONiC
▪Facebook - FBOSS
▪Ngenix (CDN) - Linux/switchdev
▪Russia Biggest Bank (Cloud) -
Linux/siwtchdev
▪…
▪Vendor Support
▪SONiC - Mellanox, Dell, Edge-Core,
Arista, Cisco, … 20+
▪Linux/switchdev - Linux, Mellanox,
Cumulus, ALT …
▪Feature Richness
▪L2 bridging
▪Linux bridge, MSTPd
▪L3 routing
▪Open OSPF/BGP implementations - 20+ yrs
▪Quagga/FRR/Bird/…
▪Tunneling - GRE/VXLAN
▪Linux kernel, OVS
▪EVPN - FRR/GoBGP
▪Security/Isolation - ACL, VRF
▪iptables, Linux TC, Linux NS/VRF
▪Management, monitoring
▪ SNMPd, hsFlowd, Grafana

27. © 2019 Mellanox Technologies | Confidential 27
Can You Afford to Lose Packets in Your Data Center?
5.2
8.4
9.6 9.7
0.3
0.9 1.0 1.1
64B 512B 1.5B 9KB
MaxBurstSize(MB)
Packet size
Microburst Absorption Capability
Spectrum Tomahawk
50
60
70
80
90
100
64 128 164 200 1518
50
60
70
80
90
100
64 128 164 200 1518
Packet Size (Bytes)Packet Size (Bytes)
Broadcom Spectrum
Microburst Absorption Fairness Avoidable Packet Loss
Broadcom Spectrum
www.Mellanox.com/tolly
www.zeropacketloss.com
25GbE to 25GbE Latency Test
Results

28. © 2019 Mellanox Technologies | Confidential 28
Power of openness

29. © 2019 Mellanox Technologies | Confidential 29
Thank You