rtnetlink
•
4 likes•2,967 views
The slides used in the Linux kernel reading party #116 in Japan. https://kernel.doorkeeper.jp/events/21043
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to rtnetlink
Similar to rtnetlink (20)
Recently uploaded
Recently uploaded (20)
rtnetlink
- 2. @takufukushima • I just worked on rtnetlink library for Java/Scala and its application • I’m not a kernel expert :-p
- 3. Agenda 1. What is Netlink and rtnetlink 2. MidoNet and Netlink 3. Enter rtnetlink 4. Wrap-up
- 4. 1. What is Netlink and rtnetlink?
- 5. Netlink as an IPC • Netlink is an intra-kernel messaging system • Netlink is an IPC between the Linux kernel and the userspace that has: • Socket interface (AF_NETLINK family with various protocols) • Broadcast messages (notifications) from the kernel triggered by other processes
- 6. History of Netlink • It was introduced in Linux 2.2, 1999, by Alexey Kuznetsov in INR RAS as a successor of ioctl for the networking interfaces • In 1995, Linux 1.3 had /dev/netlink (Skiplink; obsolete) by Alan Cox • Generic Netlink was supported in 2.6.15, 2006
- 7. Netlink use cases • iproute2, a.k.a. ip command • by Alexey Kuznetsov and Stephen Hemminger • Open vSwitch (OVS) • e.g., the communication between the datapath in the kernel and ova- vswitchd in the userspace
- 8. Netlink protocols OVS datapath • Link • Address • Route • Neighbor • Rule • QDisc • Traffic Class • Traffic Filter • … iproute2 (a.k.a. ip) Netlink multiplexer include/uapi/linux/netlink.h rtnetlink
- 10. • man 7 netlink • Netlink Protocol Library Suite (libnl) • http://www.carisma.slowglass.com/~tgr/libnl/ • RFC 3549 • https://tools.ietf.org/html/rfc3549 • Linux kernel and iproute2 source code Netlink documentations
- 11. Linux source code • Use cscope, global or your preferred tag system • net/netlink/*.[ch] • include/linux/{genetlink, netlink, rtnetlink}.h • include/net/{genetlink, netlink, rtnetlink}.h • include/uapi/linux/*.h
- 12. iproute2 source code • Use cscope, global or your preferred tag system • ip/*.[ch] • ip/include/linux/*.h • e.g., ip/include/linux/ip_link*.h for links • lib/libnetlink.c
- 13. Debugging Netlink • nltrace • https://github.com/socketpair/nltrace • nlmon & (tcpdump | netsniff-ng)
- 16. Notes on Netlink • Netlink data should be transferred in the native endian • Little endian on the little endian system • Big endian on the big endian system • You need to subscribe some groups to get the notifications from the kernel
- 19. MidoNet 101 Open vSwitch datapath in_port=1, action= ... in_port=2, action= ... . . . in_port=29, action= ... FlowTable MidoNet Agent 3. Packet execute Flow Matched Packets Unmatched Packets 1. Upcall 2. Set Flow Table Entry Userspace Kernel Host
- 20. NSDB NSDB NSDB Private Network Host Midol man Cache Datapath VM VM VM Flow Table Nova compute MidoNet APINova API Horizon MidoNet CLI Neutron API MidoNet Plugin Clients / Users Host Midol man Cache Datapath VM VM VM Flow Table Nova compute BGP Gateway Midol man Datapath Flow Table BGP Gateway Midol man Datapath Flow Table GRE/VXLAN Tunneling Internet
- 21. Midolman (MidoNet agent) NSDB NSDB NSDB Open vSwitch Datapath IF IF Interfaces on the host IF VM VM VM Midolman (MidoNet agent) Network Flow Table Watch/modify Add/remove flows Host Cache + local state Store virtual topology information Nova compute
- 22. Open vSwitch Datapath IF IF e host IF VM VM VM Midolman (MidoNet agent) Flow Table Watch/modify Add/remove flows Host Cache + local state Nova compute
- 23. Open vSwitch Datapath IF IF e host IF VM VM VM Midolman (MidoNet agent) Flow Table Watch/modify Add/remove flows Host Cache + local state Nova compute Open vSwitch Datapath IF IF e host IF VM VM VM Midolman (MidoNet agent) Flow Table Watch/modify Add/remove flows Host Cache + local state Nova compute Netlink
- 24. MidoNet speaks Netlink • MidoNet agent drives OVS datapath kernel module • MidoNet agent communicates with the kernel through Netlink • e.g., upcalls and flow installations/ invalidations
- 25. Upcall Lifecycle 1. Input stage • Get upcalls with packets from the datapath 2. Packet processing stage 1. Deduplicate and queue packets 2. Simulate packets on the virtual topology 3. Deal with the wildcard flows 4. Determine the egress physical port 3. Output stage • Emit packets and install flows based on the sims Netlink
- 26. Module diagrams
- 27. F PacketsEntryPoint NetlinkCallback Dispatcher DeduplicationActor PacketWorkflow UpcallDatapath ConnectionManager One-to-Man One-to-On HTB Supended Packets Waiting Room (NetlinkInputChannel) NetlinkChannel Fast Path State Management Open vSwitch Datapath Flow Table UpcallPacket 1. Input stage Select Loop
- 28. Datapath Controller Flow Controller PacketsEntryPoint NetlinkCallback Dispatcher DeduplicationActor PacketWorkflow UpcallDatapath ConnectionManager One-to-Many One-to-One HTB Supended Packets Waiting Room (NetlinkInputChannel) NetlinkChannel (NetlinkOutputChannel) DatapathChannel ment path Open WildcardFlow Pa Wildcard Flows Flow Managem 2. Packet processing stage PacketContext PacketContextPacketContext PacketContext PacketContext PacketContext Routing by hashing with FlowKey
- 29. Datapath Controller Flow Controller PacketsEntryPoint VirtualTo PhysicalMapper DeduplicationActor PacketWorkflow Disruptor Ring Buffer Supended Packets Waiting Room el) (NetlinkOutputChannel) DatapathChannel Flow Table DatapathReady WildcardFlow validation y Tag Virtual Topology State data / Messages Packet Flow Wildcard Flows DatapathReady Datapath port operations Flow Management 2. Packet processing stage Local datapath management • Create local datapath ports • Track UUID to port # mapping • Manage overlay tunnels PacketContext
- 30. Datapath Controller Flow Controller PacketsEntryPoint VirtualTo PhysicalMapper DeduplicationActor PacketWorkflow Disruptor Ring Buffer Supended Packets Waiting Room (NetlinkOutputChannel) DatapathChannel Flow Table DatapathReady WildcardFlow n Virtual Topology State data / Messages Packet Flow Wildcard Flows DatapathReady Datapath port operations Flow Management 2. Packet processing stage Flow Flow Flow Flow Query statistics Invalidate flows PacketContext Flow management • Cache flows • Invalidate flows when the virtual topology changed • Add flow to the datapath
- 31. Datapath Controller Flow Controller PacketsEntryPoint NetlinkCallback Dispatcher DeduplicationActor PacketWorkflow One-to-Many One-to-One B Disruptor Ring Buffer Supended Packets Waiting Room (NetlinkOutputChannel) DatapathChannel Open vSwitch Datapath Flow Table WildcardFlow Virtual Topology State data / Messages Packet Flow Wildcard Flows Datapath port operations Flow Management 3. Output stage Select Loop
- 32. Netlink/rtnetlink in MidoNet • odp library to communicate with OVS datapath • Hard-coded ip command • InterfaceScanner scans interface information on the host • e.g., interface type, MTU, …
- 34. 3. Enter rtnetlink Kill InterfaceScanner Retrived from “The Terminator”. © Hemdale Film Corporation and Orion Pictures
- 35. InterfaceScanner • Scans interface information on the host periodically • Exposes the subscription interface • e.g., Notify the current status of all interfaces to other components
- 36. New InterfaceScanner • Gets the notifications from the kernel through rtnetlink for the updates • Exposes the subscription interface • e.g., Notify the current status of all interfaces to other components
- 38. Do you know what ip a does? 1. Retrieve link information 2. Retrieve address information 3. Combine them into the single representation format 4. Display the result
- 40. Link
- 41. Addr
- 42. Do you know what ip a does? 1. Retrieve link information 2. Retrieve address information 3. Combine them into the single representation format 4. Display the result Blocking operation from the! perspective of Midolman
- 43. Now everything is asynchronous. How can we coordinate them?
- 48. 4. Wrap-up
- 49. Good bye, old InterfaceScanner Retrieved from “Terminator 2: Judgment Day”. © Carolco Pictures, Lightstorm Entertainment, Le Studio Canal+, and TriStar Pictures
- 50. rtnelink library • Retrieve/create rtnetlink resources • Observer or Subject consumes retrieved data • Coordinate async operations with RxJava • map ByteBuffer to rtnetlink resource • filter some resources • zip few different resources
- 51. New InterfaceScanner • Get the notifications from the kernel • Update the single representation format • Let Observers consume the data
- 52. Acknowledgements • The following people helped me a lot • Takayuki Usui, Guillermo Ontañón, Duarte Nunes and Ivan Kelly • Special thanks to Antoni Segura Puimedon and Hugo Benichi
- 53. (Non-academic) References • The Netlink protocol: Mysteries Uncovered , Jan Engelhardt, 2010 • http://inai.de/documents/Netlink_Protocol.pdf • Understanding And Programming With Netlink Sockets, Neil Horman, 2004 • http://people.redhat.com/nhorman/papers/netlink.pdf • Netlink - Wikipedia, the free encyclopaedia • http://en.wikipedia.org/wiki/Netlink
- 54. This is the end of slides. Any questions?