Introduction to ONIE, the Open Network Install Environment and the new world of whitebox switches.

1. Open Network Install Environment
NETWAYS OSDC 2014, Berlin
Nat Morris
9th April 2014

2. cumulusnetworks.com 2
•Overview
•What Is ONIE?
•Lessons Learned
•ONIE Development
•Demo
Agenda

3. IP Fabric Networking Landscape
cumulusnetworks.com 3
Network
Hardware
NetworkOS
Open Closed

4. The Expanding Landscape
hardware
operating system
appapp
hardware
operating system
app app
Single Vendor Blob Multi-Vendor
Ecosystem
app app
cumulusnetworks.com 4

5. Understanding Characteristics of a Leaf Switch
5cumulusnetworks.com
10/40 Gigabit
spine uplink ports
Serial
console port
Ethernet Out-of-
Band
Management Port
* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options
* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options

6. Understanding Characteristics of a Spine Switch
6cumulusnetworks.com
Serial
console port
Ethernet Out-of-
Band
Management Port
* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options

7. Bare Metal Switch Provisioning
Similar approach to installing OS on server
 BIOS + PXE = U-Boot + ONIE (Open Network Install
Environment)
 Supported hardware (HCL) preloaded with ONIE
 ONIE available on GitHub
• http://onie.github.io/onie/
cumulusnetworks.com 7
bare metal server
operating
system
app app app
BIOS and PXE
bare metal switch
operating
system
app app app
U-Boot and ONIE

8. Choice
cumulusnetworks.com 8

9. Choice
cumulusnetworks.com 9

10. What Is ONIE?
Network OS Install Environment
 Provides an environment for network OS installer
discovery and execution
 Like a pre-installed kickstarter
 Defined by its behaviors
 Implemented using a modern Linux kernel and BusyBox
An Open Source Project within OCP
 http://www.onie.org/
Evolving …
cumulusnetworks.com 10

11. Hardware Vendors
cumulusnetworks.com 11

12. Operating System Vendors
cumulusnetworks.com 12

13. ONIE: Anatomy of a Network Switch
cumulusnetworks.com 13
( Management Interfaces ) ( Data Plane )
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switchin
g
ASIC
Serial
Console
Ethernet
Mgmt Port
10Gb
Port
40Gb
Port…
10Gb
Port
40Gb
Port
…
PCIe

14. ONIE: Uses Management Interfaces
cumulusnetworks.com 14
( Management Interfaces ) ( Data Plane )
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switchin
g
ASIC
Serial
Console
Ethernet
Mgmt Port
10Gb
Port
40Gb
Port…
10Gb
Port
40Gb
Port
…
PCIe

15. ONIE: Bare Metal Install – First Time Boot Up
cumulusnetworks.com 15
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Installer
(OS Vendor)
Boot Loader
• Low Level boot loader, configures CPU complex
• Loads and boots ONIE
ONIE
• Linux Kernel with Busybox
• Configures management Ethernet interface
• Locates and executes an OS installer
• Provides tools and environment for installer
OS Installer
• Available from network or USB
• Linux executable
• Installs vendor OS into mass storage
Network OS
(OS Vendor Supplied)
Fetches
Installs

16. ONIE: Subsequent Reboots – Vendor’s OS is Already
Installed
cumulusnetworks.com 16
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Boot Loader
• Low Level boot loader, configures CPU complex
• Loads and boots OS vendor’s installed OS
Network OS
• Configures Switching ASIC
• Runs Network Protocols
• Provides CLI
Network OS
(OS Vendor Supplied)
ONIE
• Still exists, but is not used
• Available for uninstall / re-install operations

17. ONIE: Network OS Installer Discovery and Install
Behavior
cumulusnetworks.com 17
Configure Network
Interface
Locate Installer
Run Installer
• Uses DHCPv4, DHCPv6
• Configures Ethernet interface for IPv4 / IPv6
• Configures DNS and hostname
• Determines the location of an installer executable
• Examines local file systems, e.g. USB flash drives
• Uses DHCP options, DNS Service Discovery, Multicast
DNS and IPv6 Neighbors
• Downloads installer via URL
• Passes various environment variables to installer
• Launches installer

18. Why Not Use PXE?
cumulusnetworks.com 18
• IPv6 and HTTP out of the box
• Use existing Linux device drivers. No need to write new ones
• Integrated automation

19. ONIE: Other Behaviors
Provides a mechanism for a network OS to
invoke the above behaviors.
cumulusnetworks.com 19
Reinstall Remove currently installed OS and return to
the “out of box” provisioning state
Uninstall Completely wipe out everything, except ONIE
Rescue Reboot box into ONIE for repair, debug and
forensics
Update Install a new ONIE version
Diag Run HW Vendor’s diag (optional)

20. Lessons Learned
Hardware Platforms
 Diverse mix of CPU complex designs, even within a
single CPU family.
 HW Vendors need freedom to customize ONIE
 Need mechanism to run HW vendor diag
Recurring Themes
 TLV based EEPROM format widely adopted
 Common HW designs reduce development time
cumulusnetworks.com 20

21. ONIE Ongoing Development
Support Multiple CPU Architectures
 PowerPC – Today
 x86 – VM Prototype available today
 Thinking about ARM, MIPS
 Maintain ONIE behaviors across architectures
Testing and Compliance
 Enhance and extend regression test suites
 Develop compliance test suites
cumulusnetworks.com 21

22. ONIE Ongoing Development
New Features
 DHCPv6
 DNS Service Discovery / Multicast DNS discovery
 HW Vendor diagnostic
Releases
 Quarterly release cadence
 Ongoing maintenance
 Enhancements
cumulusnetworks.com 22

23. ONIE x86 Strategy
 Use existing BIOS from hardware vendors
 During manufacturing install GRUB2 and ONIE-x86
on the mass storage block device
 NOS Installer adds partitions, installs software and
updates GRUB2 configuration.
cumulusnetworks.com 23

24. Block Device Partitioning
 Partition using GUID Partition Table (GPT) format
 GPT supported by Linux, GRUB, UEFI
 Supports dual-booting multiple operating systems
cumulusnetworks.com 24
Partition # Name R/W Notes
1 GRUB Boot r/o Used by GRUB and
GPT
2 ONIE BOOT r/o ONIE Kernel
3 ONIE CONFIG r/w Configuration
4+ For NOS use

25. GRUB Menu
cumulusnetworks.com 25

26. NOS Installer Duties
 Create new GPT partition(s)
 Create file systems on partition(s)
 Install NOS files into partition(s)
 Update ONIE-CONFIG using ONIE provided CLIs
 onie-boot-entry-add
 onie-boot-entry-remove
 onie-boot-entry-show
 onie-boot-default
 onie-boot-update
cumulusnetworks.com 26

27. Resources
ONIE Websites
 Main Page: http://www.onie.org/
 Source Code: https://github.com/onie/onie/
 Documentation: http://onie.github.io/onie/
cumulusnetworks.com 27

28. © 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus
Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is
used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
Thank You!
cumulusnetworks.com 28

29. Back Up Slides – Cumulus Background
cumulusnetworks.com 29

30. Pushing Changes Down
cumulusnetworks.com 30
CPU, RAM, Flash, etc. Switch Silicon
Front Panel Ports
lldpd
Routing Tables
ARP
Table
Devices
Bridge FDB Filter Tables
Bonds VLANs
LinuxKernel
Virtual Kernel Ports
Bridging
mstpd
ACLRouting Suite
Quagga
snmpd
vconfig
iptable
ebtable
ip6tableiproute2
VXLAN
Bridges
Switch HAL
brctl
Switch
Driver
UserSpace
Quagga daemon,
Quagga.conf, and vtysh
CLI and
/etc/network/interfaces
switchd

31. Cumulus Linux Hardware Compatibility List
Cumulus Networks and Partner Confidential
40G Portfolio Part number Description
Merchant
Silicon
Cumulus Linux Release
Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later
Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later
Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later
Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later
10G Portfolio Part number Description
Merchant
Silicon
Cumulus Linux Release
Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later
Dell S4810 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.x or later**
Edge-Core AS5600-52X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Edge-Core AS5710-54X 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
Edge-Core AS5710-96X 96 x 10G-SFP+ and 8 x 40G-QSFP+ Trident II 2.1 or later**
Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later
Quanta QCT T-3048-LY2* 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Quanta QCT T5048-LY8* 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
1G Portfolio Part number Description Merchant Silicon Cumulus Linux Release
Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later
Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later
Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later
31