The document discusses the evolution of OSv from running on BitVisor (OSb) to running directly on hardware (OSp). It provides details on implementing a 10GbE network driver for the Intel X540 NIC in OSp. The driver implementation covers initialization, TX and RX operations using descriptor rings, and the interface with OSv. Performance is improved over OSb but there is still work needed on interrupts and advanced features. Additional scripts are discussed that allow OSv to directly access PCI devices when running on KVM/Linux for further optimization.

1. OS*: OSv on *
@ysmoo

2. BitVisor
BitVisor
Hardware
OS
App App App
http://www.justis.as-1.co.jp
https://bitbucket.org/bitvisor/bitvisor

3. OSv
VMM
Hardware
OSv
App
http://medical-care.feed.jp
https://github.com/cloudius-systems/osv

4. BitVisor

5. BitVisor OSv

6. BitVisor OSv
http://www.root.ne.jp/nishide/shs/

7. OSv on BitVisor
BitVisor
Hardware
OSv
App

8. OSv on BitVisor
BitVisor
Hardware
OSv
App
OSb

9. OSb Design
BitVisor
Hardware
OSv
App
Virtio NIC
With a small fix,
OSv can handle chipsets
OSv has an AHCI driver
BitVisor providesVirtio NIC

10. OSb Design
BitVisor
Hardware
OSv
App
Virtio NIC
With a small fix,
OSv can handle chipsets
OSv has an AHCI driver
BitVisor providesVirtio NIC

11. One thing to fix
Skipped the code!
(drivers/acpi.cc:early_init())

12. OSb Design
BitVisor
Hardware
OSv
App
Virtio NIC
With a small fix,
OSv can handle chipsets
OSv has an AHCI driver
BitVisor providesVirtio NIC

13. Para Pass-through
Virtio
BitVisor
Hardware
Virtio Ring
PRO/1000 Ring
OSv
App
MSI-X etc.
Handle minimal operations.
PCI Function
Pass-through!
For more details, see:
http://www.slideshare.net/yushiomote/osb-osv-on-bitvisor-2

14. for (12-billion times) val++;
0.00
6.50
13.00
19.50
26.00
OSb Linux (Host, Bare-metal)
25.51
20.75
ElapsedTime(sec)

15. Netperf TCP_STREAM
0.00
237.50
475.00
712.50
950.00
OSb Linux (Host, Bare-metal)
941.39921.83
Throughput(Mbps)

16. Ping
0.00
0.04
0.08
0.12
0.16
OSb Linux (Host, Bare-metal)
0.16
0.12
Latency(msec)

17. Thinner and thinner
Hardware
KVM/Linux
OSv
Hardware
BitVisor
OSv12.8M LOC
0.3M LOC
virtio
sched.
driver
http://cdn-ak.f.st-hatena.com/images/fotolife/j/jovi1kamiya/20140607/20140607114847.jpg

18. For
ultimate thinness…

19. OSv on Physical Machine
Hardware
KVM/Linux
OSv
Hardware
BitVisor
OSv
Hardware
OSv
OSp

20. OSp Design
Hardware
OSv
App
Physical NIC
Driver
Implemented
Intel X540 10GbE Driver
in C++11
With a small fix,
OSv can handle chipsets
OSv has an AHCI driver

21. About implementation
• Intel NIC TX/RX Basics
• Device initialization
• TX initialization/operations
• RX initialization/operations

22. Intel NIC TX/RX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers

23. Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers

24. Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers

25. Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers
Xmit!

26. Intel NIC RX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Consumer I/O Register Producer I/O Register
OSv
I/O Registers
Recv!

27. Device Init.

28. TX Init.

29. TX Interface
(OSv => Driver)
• OSv’s upper layer (TCP/IP) calls:
• int try_xmit_one_locked(void *mbuf_etc)
• Then, driver puts mbuf on ring buffer.
• bool kick_hw()
• Then, driver updates producer reg.
• …
EOP EOP
mbuf

30. try_xmit_one_locked()
to put mbuf on ring buffer

31. kick_hw()
to update producer reg.

32. RX Init.

33. RX Interface
(Driver => OSv)
• struct ifnet *_ifn = if_alloc (IFT_ETHER);
• On init., driver creates ifnet and…
• ether_ifattach(_ifn, macaddr);
• …passes it to the upper layer. (Then, the upper
layer puts an callback function on ifnet.if_input)
• (*_ifn->if_input)(_ifn, mbuf);
• On packet reception, driver calls the callback
function (if_input) by passing mbuf as an
argument.

34. Under improvement
(A lot of TODOs)
• No interrupts… (bad polling impl.)
• No TSO, LRO, checksum assist…
• No advanced descriptor… (legacy
descriptor seems slow)
• …

35. Netperf TCP_STREAM
0.00
2000.00
4000.00
6000.00
8000.00
OSp OSv
7229.00
4200.00
Throughput(Mbps)

36. Future Work
0.00
2000.00
4000.00
6000.00
8000.00
OSp OSv
7229.00
4200.00
Throughput(Mbps)
Revenge of the OSp

37. One more thing
KVM/Linux
Hardware
OSv
App
./scripts/bindctrl bind 8086 1528 0000:07:00.0
./scripts/run.py -a 07:00.0
Additional scripts & options
to start OSv with
PCI pass-through config.

38. Output
$ sudo ./scripts/run.py -n -v -c 8 -a 07:00.0
ixgbe [00:04.0] 8086:1528
Address: a0:36:9f:14:5f:84
PHY Status:
State: Connected
Type: MDI-X
Speed: 10Gbps
Duplex: Full
MAC Status:
Link: UP
Speed: 10Gbps
ixgbe [00:04.0] 8086:1528
eth0: 192.168.122.15
ixgbe0: 192.168.100.2
/#
/#
/#
/# exit
Goodbye

39. Summary
OSv → OSb → OSp

40. Thank you !
http://www.root.ne.jp/nishide/shs/

41. References
• The Ugly Patch of Intel X540 10GbE Driver
and Additional scripts for PCI Pass-through
Configuration
• http://goo.gl/0yRcV4
• git diff -r 5e5c227b2aec23418e5e3fd104cc21e98f31729c
• Usage & True Motivation
• http://www.yushiomote.org/blog/?p=428