Johann Tönsing, Chief Architect & SVP, Software, presentation from OpenStack Days: Silicon Valley 2016: "Using Agilio SmartNICs for OpenStack Networking Acceleration."

1. © 2016 NETRONOME
Johann Tönsing
August 9, 2016
Using Agilio™ SmartNICs for
OpenStack Networking Acceleration

2. © 2016 NETRONOME 2
Agenda
• Overview of Traditional OpenStack Networking Options
OVS (with/without connection tracking), Contrail vRouter, SR-IOV, VirtIO…
• Issues: Flexibility, Performance, CPU Utilization
• Accelerating and Offloading OpenStack Networking using Agilio™ SmartNICs
• Results: Features, Throughput, CPU Utilization
• Implementation Details: Hardware and Software Architectures
• Evolution of OpenStack Networking
• Supporting P4, C and eBPF Programmability
• Integration Activities
• Conclusions

3. © 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /
Virtual Switching
Technology
OpenStack
Control Plane
Forwarding Overlays (Tunnels) Security Groups
(Microsegmentation,
Stateful Firewalling)
SR-IOV with
regular NIC
Vendor specific plugin Limited
E.g. MAC/VLAN
based directing
None None

4. © 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /
Virtual Switching
Technology
OpenStack
Control Plane
Forwarding Overlays (Tunnels) Security Groups
(Microsegmentation,
Stateful Firewalling)
SR-IOV with
regular NIC
Vendor specific plugin Limited
E.g. MAC/VLAN
based directing
None None
OVS
ML2 plugin
(optional: SDN controller)
OVN plugin
L2 / L3 VXLAN, GRE,
(for OVN) GENEVE
…
Emerging:
NSH (for NFV)
OVS 2.5+ can interface
to Linux conntrack
(Previously Linux bridge
with iptables/nftables)

5. © 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /
Virtual Switching
Technology
OpenStack
Control Plane
Forwarding Overlays (Tunnels) Security Groups
(Microsegmentation,
Stateful Firewalling)
SR-IOV with
regular NIC
Vendor specific plugin Limited
E.g. MAC/VLAN
based directing
None None
OVS
ML2 plugin
(optional: SDN controller)
OVN plugin
L2 / L3 VXLAN, GRE,
(for OVN) GENEVE
…
Emerging:
NSH (for NFV)
OVS 2.5+ can interface
to Linux conntrack
(Previously Linux bridge
with iptables/nftables)
Contrail vRouter
Contrail plugin (via
Contrail controller)
L2 / L3 MPLS in UDP,
MPLS in GRE,
VXLAN
…
Built in, via flow table

6. © 2016 NETRONOME
Traditional OpenStack Networking Options
3
Forwarding /
Virtual Switching
Technology
OpenStack
Control Plane
Forwarding Overlays (Tunnels) Security Groups
(Microsegmentation,
Stateful Firewalling)
SR-IOV with
regular NIC
Vendor specific plugin Limited
E.g. MAC/VLAN
based directing
None None
OVS
ML2 plugin
(optional: SDN controller)
OVN plugin
L2 / L3 VXLAN, GRE,
(for OVN) GENEVE
…
Emerging:
NSH (for NFV)
OVS 2.5+ can interface
to Linux conntrack
(Previously Linux bridge
with iptables/nftables)
Contrail vRouter
Contrail plugin (via
Contrail controller)
L2 / L3 MPLS in UDP,
MPLS in GRE,
VXLAN
…
Built in, via flow table
Additional features: load balancing, NAT etc.

7. © 2016 NETRONOME
OpenStack Networking Options Evaluated
4
Forwarding /
Virtual Switching
Technology
Traditional Approach Agilio™ SmartNIC Accelerated Approach
SR-IOV
Limited expressiveness to direct traffic to
VMs (no support for general match/action
rules, tunnel termination, stateful firewalling)
High throughput
No VM migration support
Full OVS or vRouter virtual switching incl.
tunnel termination, stateless/stateful firewalling
and SR-IOV based data delivery to VMs
High throughput
VirtIO integration, supporting VM migration
OVS
and
Contrail vRouter
High expressiveness - match/action,
tunnels, stateless/stateful firewalling etc.
Limited throughput
High CPU utilization (e.g. 50% of cores)
Same expressiveness - match/action, tunnels,
stateless/stateful firewalling etc.
Higher throughput (~5x higher)
Lower CPU utilization (~10x lower)

8. © 2016 NETRONOME 5
Agilio™ CX SmartNIC Family
• Optimized for standard server based cloud data centers
• Low Profile Half Length PCIe form factor, power < 25W
• Based on Netronome’s NFP-4xxx silicon (72 C programmable cores, 8 threads each)
• 2GB DRAM for lookup tables / state tables (millions of entries)
• Dataplane fully implemented in software
1x 40GbE 2x 40GbE2x 10GbE New: 2x 25GbE
Also available: Agilio™ LX 2x40G / 1x100G with dual PCIe interfaces, 120 cores, 8GB DRAM…

9. © 2016 NETRONOME 6
OVS Throughput vs. Packet Size
OVS L2 Forward to VMs
Packet Size
MillionsofPacketsperSecond
OVS VXLAN + L2 Forward to VMs
Packet Size
MillionsofPacketsperSecond

10. © 2016 NETRONOME 7
OVS Throughput vs. Number of Rules
5
10
15
20
25
30
OVS in Kernel
Space
OVS in User Space
on DPDK
100 Wildcard Rules
1000 Wildcard Rules
10000 Wildcard Rules
64000 Wildcard Rules
MillionsofPacketsperSecond
12 CPU Cores
12 CPU Cores
OVS Offloaded to
Agilio™ CX-4000
1 CPU Core
5X Throughput Improvement + 90% CPU Savings
OVS L2/L3 Forwarding to 8 VMs with 64K Flows

11. © 2016 NETRONOME 8
Tested Scenario: Server CPU Core Allocation
Unaccelerated OVS (Kernel / User Mode)
Agilio™ OVS
OVS

12. © 2016 NETRONOME 9
Efficiency: Throughput Per Server CPU Core
Throughput with single server CPU core
dedicated to network related processing
MillionsofPacketsperSecond
• 50x Efficiency vs. Kernel OVS
• 20x Efficiency vs. User Mode (DPDK) OVS
=> Replace 3-6 racks with 1 rack!
Throughput / efficiency similar for
Agilio™ Contrail vRouter
Benefit for your use case:
search for
“netronome.com roi calculator”

13. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute
Action
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act

14. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act

15. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss

16. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

17. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

18. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

19. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
4 Flow tracking: per-microflow state learning
4
Self Learning
Exact Match
Flow Tracker
Miss
Hit
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

20. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
Conn
track
FTP
SIP
4 Flow tracking: per-microflow state learning
4
Self Learning
Exact Match
Flow Tracker
Miss
Hit
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

21. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
Conn
track
3 Offload datapath: copy match tables, sync stats
3
Conn
track
FTP
SIP
4 Flow tracking: per-microflow state learning
4
Self Learning
Exact Match
Flow Tracker
Miss
Hit
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

22. © 2016 NETRONOME
Offload Model: Agilio™ OVS Acceleration
10
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
Conn
track
3 Offload datapath: copy match tables, sync stats
3
5 Offload connection tracking: synchronize state
5
Conn
track
FTP
SIP
4 Flow tracking: per-microflow state learning
4
Self Learning
Exact Match
Flow Tracker
Miss
Hit
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

23. © 2016 NETRONOME
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
vRouter
Forwarding
vRouter
Forwarding

24. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
vRouter
Forwarding
vRouter
Forwarding

25. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
vRouter
Forwarding
vRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied

26. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
Miss
vRouter
Forwarding
vRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied

27. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
Miss
Miss
vRouter
Forwarding
vRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied

28. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute
Action
vRouter
Flow
Classify
Miss
Miss
vRouter
Forwarding
vRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied

29. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute
Action
4 Offload flows: copy tables, sync stats
4
vRouter
Flow
Classify
Miss
Miss
vRouter
Forwarding
vRouter
Forwarding
2
2 Offload forwarding: pre-emptively copied

30. © 2016 NETRONOME
vRouter Datapath
vRouter
Flow
Classify
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
vRouter
Forwarding
vRouter
Forwarding
Agilio™ vRouter Acceleration
11
vRouter
Subsystem
vRouter
Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
vRouter
CLI
1 Configuration via controller or CLI: BGP over XMPP
3
3 Userspace agent populates kernel tables
(Nova, Neutron)
Execute
Action
4 Offload flows: copy tables, sync stats
4
vRouter
Flow
Classify
Miss
Miss
vRouter
Forwarding
vRouter
Forwarding
Notes:
- Control protocol: BGP over XMPP
- Tunnel protocols: VXLAN, MPLS over UDP / GRE
- Distinct policy (flows - ACLs) and forwarding (IP/MPLS)
- Flow table is used to implement stateful firewalling
2
2 Offload forwarding: pre-emptively copied

31. © 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
• SmartNIC with dynamically downloadable
firmware

32. © 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
• SmartNIC with dynamically downloadable
firmware
Contrail OVS
OpenStack ONOS ODL
Linux BSD
• OVS / Contrail / Linux eBPF datapath on host
can be accelerated by SmartNIC
D
P
D
K
eBPF

33. © 2016 NETRONOME
SmartNIC Firmware: Pre-programmed or Custom
Compiler Debugger
Run-Time
app.P4 app.C
Editor
• Firmware can be developed in P4 and/or C
• TBD how to integrate custom programmed
datapaths (implemented in P4 / C etc.) into
OpenStack
• SmartNIC with dynamically downloadable
firmware
Contrail OVS
OpenStack ONOS ODL
Linux BSD
• OVS / Contrail / Linux eBPF datapath on host
can be accelerated by SmartNIC
D
P
D
K
eBPF

34. © 2016 NETRONOME 13
SmartNIC Datapath “Worker” Software Architecture
• Load balancer distributes each packet to next available thread for optimum throughput
• Hardware assisted reordering ensures packet order is maintained
• Flow tracker statefully learns / tracks millions of sessions
• Matching performed using DRAM-backed tables - capacity > 500k entries
• Actions efficiently performed in on-chip memory
Parse Match Act
Flow Tracker
Learn microflows
Cache action
Datapath
Run to completion
Load
Balance
C Plugin
Re-
order
Pool of worker threads on flow processing cores
net or
PCIE
net or
PCIE
= Ring / Work Queue (multi producer / consumer)

35. © 2016 NETRONOME 14
Example: P4 “main” implementing a simple NIC
header_type eth_hdr {
fields {
dst : 48;
src : 48;
etype : 16;
}
}
header eth_hdr eth;
parser start {
return eth_parse;
}
parser eth_parse {
extract(eth);
return ingress;
}
action drop_act() {
drop();
}
action fwd_act(port) {
modify_field(standard_metadata.egress_spec,
port);
}
table in_tbl {
reads {
standard_metadata.ingress_port : exact;
}
actions {
fwd_act;
drop_act;
}
}
control ingress {
apply(in_tbl);
}

36. © 2016 NETRONOME
SmartNIC Firmware (P4/C)
Example of Fully Customized Datapath (P4 / C)
15
Run-Time
Interface
Server (x86 - Linux)
PCIe
Agilio™ SmartNIC
Virtual Machine 1
VNF
Kernel
Mode
(C)
netdev
P4 / C
Development
Environment
Edit - Debug
Control
App
Populate
tables,
display
statistics
Security
µVNF (C)
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Virtual Machine 2
VNF
User
Mode
(C)
DPDK
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Match
Protocol
Meter
Other
TCP
Concepts:
• P4 and C running on SmartNIC
implements datapath - e.g.
defines protocols, match / action
behavior
• Datapath steers traffic to VNFs
running on x86 server and
on SmartNIC

37. © 2016 NETRONOME
SmartNIC Firmware (P4/C)
Example of Fully Customized Datapath (P4 / C)
15
Run-Time
Interface
Server (x86 - Linux)
PCIe
Agilio™ SmartNIC
Virtual Machine 1
VNF
Kernel
Mode
(C)
netdev
P4 / C
Development
Environment
Edit - Debug
Control
App
Populate
tables,
display
statistics
Security
µVNF (C)
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Virtual Machine 2
VNF
User
Mode
(C)
DPDK
Timestamp
µVNF (C)
Latency Stats
µVNF (C)
Match
Protocol
Meter
Other
TCP
Concepts:
• P4 and C running on SmartNIC
implements datapath - e.g.
defines protocols, match / action
behavior
• Datapath steers traffic to VNFs
running on x86 server and
on SmartNIC
To Discuss:
OpenStack
Integration

38. © 2016 NETRONOME
Extending OpenStack Networking - P4/C Plugins
16
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVS
Kernel DP
Match/Act
OVS
Kernel DP
Match/Act
Miss
Miss

39. © 2016 NETRONOME
Extending OpenStack Networking - P4/C Plugins
16
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
1 Configuration via controller, CLI, or Callable API
2
2 OVS userspace agent populates kernel cache
(Nova, Neutron)
Execute
Action
Open vSwitch Datapath
Execute Action
(e.g. Entunnel,
Deliver to VM,
Send to Port)
3 Offload datapath: copy match tables, sync stats
3
OVS
Kernel DP
Match/Act
Datapath
Extension
or Plugin
P4 / C
in Sandbox
DP Ext.
4 Datapath extension software
4
4
OVS
Kernel DP
Match/Act
Miss
Miss

40. © 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
Action
OVS
Kernel DP
Match/Act
P4
Matching

41. © 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
Action
P4 Generated Datapath
Execute
P4
Action
OVS
Kernel DP
Match/Act
P4
Matching

42. © 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
Action
P4 Generated Datapath
Execute
P4
Action
OVS
Kernel DP
Match/Act
P4
Matching
Fallback

43. © 2016 NETRONOME
OVS “on” SmartNIC P4 Datapath
17
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
Action
P4 Generated Datapath
Execute
P4
Action
OVS
Kernel DP
Match/Act
P4
Matching
Fallback
Fallback

44. © 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching

45. © 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
P4 Generated Datapath
Execute
P4 / OVS
Action
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching

46. © 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
P4 Generated Datapath
Execute
P4 / OVS
Action
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching
Fallback

47. © 2016 NETRONOME
P4 “into” OVS Datapath
18
Open vSwitch Subsystem
OVS
Agent
OpenFlow
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
OVS CLI
Callable
API
(Nova, Neutron)
P4 Generated Datapath
Execute
P4 / OVS
Action
Execute
P4 / OVS
Action
P4 / OVS
Matching
P4 / OVS
Matching
Fallback
Fallback

48. © 2016 NETRONOME
P4 “instead of” OVS (Datapath and Control)
19
Open vSwitch Subsystem
P4 Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
(Nova, Neutron)
P4 Generated Datapath
Execute
P4
Action
Execute
Action
P4
or eBPF
or XDP
P4
Matching
Fallback
Fallback
1 Protocol(s) to be discussed
(could become callable API)

49. © 2016 NETRONOME
P4 “instead of” OVS (Datapath and Control)
19
Open vSwitch Subsystem
P4 Agent
Virtual Machine
Virtual Machine
Virtual Machine
x86 Kernel
x86 Userspace
PCIe
Virtual Machine
SR-IOV /
VirtIO VFs
SR-IOV /
VirtIO VFs
Agilio™
SmartNIC
Apps
Apps
1
netdev or DPDK
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
Apps
netdev or DPDK
(Nova, Neutron)
P4 Generated Datapath
Execute
P4
Action
Execute
Action
P4
or eBPF
or XDP
P4
Matching
Fallback
Fallback
1 Protocol(s) to be discussed
(could become callable API)
Other open issues:
- Downloading programs via
OpenStack or other systems
- Scheduling VMs to run on
nodes with acceleration hardware
(Nova)
Acceleration discussion started at
OpenStack Spring 2016 (Austin)

50. © 2016 NETRONOME 20
Integration / Open Sourcing Activities
Area Activities
Linux drivers VF netdev (kernel device
driver) upstreamed in
kernel 4.5
Patches available for
older kernels, or use
VirtIO
Representative netdev
(for fallback processing
of traffic) proposals
imminent
FreeBSD drivers Kernel device driver implemented
DPDK drivers Poll mode driver
upstreamed in DPDK 2.2
Patches available for
older DPDK versions
Open vSwitch
acceleration
integration
Hooks for acceleration
proposed in October
2014
Iteration in progress
OpenStack
integration
Proposing plugins and
agents to support virtual
switching acceleration
via RFE process
Integration for OVS in
process - Mirantis,
Ericsson CEE etc.
Integration for Contrail
vRouter in process -
Juniper etc.

51. © 2016 NETRONOME 20
Integration / Open Sourcing Activities
Area Activities
Linux drivers VF netdev (kernel device
driver) upstreamed in
kernel 4.5
Patches available for
older kernels, or use
VirtIO
Representative netdev
(for fallback processing
of traffic) proposals
imminent
FreeBSD drivers Kernel device driver implemented
DPDK drivers Poll mode driver
upstreamed in DPDK 2.2
Patches available for
older DPDK versions
Open vSwitch
acceleration
integration
Hooks for acceleration
proposed in October
2014
Iteration in progress
OpenStack
integration
Proposing plugins and
agents to support virtual
switching acceleration
via RFE process
Integration for OVS in
process - Mirantis,
Ericsson CEE etc.
Integration for Contrail
vRouter in process -
Juniper etc.
Also: P4 and protocol independent forwarding, switchdev, eBPF / XDP…

52. © 2016 NETRONOME 21
Next Steps
• Use Agilio™ SmartNICs with existing dataplanes
• Use Agilio™ OVS (with / without Conntrack)
• Use Agilio™ vRouter
• Future: Agilio™ eBPF/XDP, etc.
• Program Agilio™ SmartNICs
• Use APIs (on x86 servers) - with above dataplanes
• Program in P4 and/or C (on SmartNIC / on x86)
• Improve performance + free up server resources!

53. © 2016 NETRONOME
More information: netronome.com and open-nfp.org
Thank You!
22