Today’s cloud network consists of sophisticated virtual
networks, and a virtual switch is a key element of these
networks. Although there is tremendous interest in measuring
cloud network performance, little is known about the impact
of software-based virtual network on latency. In this paper, we
conduct the impact of virtual network on latency in the public
cloud based on OpenStack. We measured the throughput of VMs
and simultaneously captured their packets on hosts. We analyzed
the traces by using well-known metrics, such as throughput and
RTT, and investigated the abrupt fluctuation of latency called as
‘the burstiness of latency’. We quantitatively clarify the impact of
software-based virtual network on latency. In our public cloud,
the latency is approximately 35.2% of RTT and 10% of burstiness
mainly contributes to the increased RTT. The total latency was
increased by the receiving side regardless of data and ACK
paths. Our analysis results, discussions, and implications can
not only help cloud researchers and developers design the next
generation of software-based virtual network but can also help
cloud operators improve the performance of virtual network.
2. Introduction
OpenStack (https://www.openstack.org)
It is OSS/ecosystem for cloud management
It has been widely used by many cloud operators
Overview of public cloud based on OpenStack
Compute Node : Deploy VMs
Network Node : Provide L3 routing via virtual router
Network fabric : Inter-connector between VMs
1 Copyright 2018 FUJITSU LABORATORIES LIMITED
3. Previous works on measuring cloud network
Many studies have focused on cloud network
performance and depended on information at VM side
Few prior works of the latency on the virtual network
One study measured the latency of virtual networks, such as Xen,
Linux, and VServer, in a local environment [Sigcomm CCR 41]
Their results are too limited to be generalized to the latency
of cloud network architecture based on OpenStack
2 Copyright 2018 FUJITSU LABORATORIES LIMITED
4. Research goal
3 Copyright 2018 FUJITSU LABORATORIES LIMITED
Understanding the latency of
software-based virtual
network on the public cloud
5. Approach
Focus on packet traces on host side
Measure VM-to-VM TCP throughput on the public cloud
Capture their packets at multiple capture points on the host
Characterize the impact of software-based virtual network in the
public cloud through quantitative analyses
4 Copyright 2018 FUJITSU LABORATORIES LIMITED
6. Target cloud based on OpenStack
This public cloud has been in production use since 2015
Compute Node
•Have 48CPU cores, 256GB RAM, and a 10Gbps NIC
•Deploy approximately 40-50 VMs
Network Node
•Provide L3 routing via virtual routers for multi tenants
Network fabric for L2 forwarding
•VLAN and MLAG are used to separate tenant traffic and all of physical
links are 10 Gbps
5 Copyright 2018 FUJITSU LABORATORIES LIMITED
7. Software-based Virtual Network
Linux bridge
•iptables on Linux bridge is
provided as a security group
•Each cloud user can freely
add entries to iptables
Open vSwitch (OVS)
•It is responsible for
external/internal
communications
6 Copyright 2018 FUJITSU LABORATORIES LIMITED
Virtual network on the compute node
Will capture
packets on
multiple points
9. Node selection
We focused on low utilization for suitable nodes
Measure CPU and memory usage, traffic volume, and the number
of VMs at each node
Consider the resource metrics as candidates for low utilization
Overview of selected nodes
There were no abrupt changes in the metrics
•We believe that it can indicate the low utilization
VMs that had one CPU core and 4GB RAM were deployed for the
measurement at different racks
•To avoid extra overhead by iptables (Linux bridge),
we did not add any rules to the security group
8 Copyright 2018 FUJITSU LABORATORIES LIMITED
Number of VMs Idle CPUs [%] Memory [%] Traffic volume [Gbps]
40 VMs (48 CPUs) 95 70 Smaller than 1Gbps*
* The traffic volume is in the night
< Resource metrics >
10. Measurement method : VM-to-VM
Overview of experiment
Measure TCP throughput using iperf3 for 30 seconds
Control write lengths (128, 256, 512, 1024, 2048, or 65535 bytes)
Capture packets at NIC (eth) and vports (tap and qvb) concurrently
•Analyze well-known metrics and Data/ACK path latency
9 Copyright 2018 FUJITSU LABORATORIES LIMITED
O : capture point
11. Packet capture overhead
Overview of the capture overhead at our measurement
Total CPU utilization for the capture : below 5%
•The compute node had 48 CPUs and three idle CPUs were used
The utilization of each CPU core : roughly 50-60%
Throughput with/without captures (length : 1024 bytes)
The throughput with the packet capture is similar to that without
the packet capture
We believe that the capture overhead would be small
10 Copyright 2018 FUJITSU LABORATORIES LIMITED
12. Write lengths
[bytes]
128 256 512 1024 2048 65535
(64 KB)
Mean throughput
[Mbps]
721 1375 2524 4389 6382 6707
TCP Throughput
Throughput statistics
We ran our experiments in the night
•The traffic volume was smaller than 1 Gbps in the night
Throughput with 2048 bytes is similar to that with 65535 bytes
•The TCP throughput with 64 KB was similar to the maximum
throughput on our cloud
While each result had different throughput, well-known metrics
were fundamentally similar
11 Copyright 2018 FUJITSU LABORATORIES LIMITED
We select the representative result
with 1024 bytes
13. RTT and latency in our measurement
RTT
Indicates the time between
sending a data packet and
receiving an ACK packet
Three RTT measurement points
•One NIC : eth
•Two vports : tap and qvb
Latency
Indicates the time consumed
by the software-based virtual
network on server
Four sections of latency
•Data sending and Data receiving
•ACK sending and ACK receiving
12 Copyright 2018 FUJITSU LABORATORIES LIMITED
14. RTT between capture points
RTT (1024 bytes)
Compare statistics of RTT with those on MS DCN (Sigcomm 2015)
Find the overhead induced by the software-based virtual network
13 Copyright 2018 FUJITSU LABORATORIES LIMITED
RTT Mean [μs] 50%tile [μs] 99%tile [μs]
tap 386 377 871
qvb 331 322 714
eth 305 300 666
MS DCN NO result 268 1340
15. Latency on virtual network
Latency (1024 bytes)
The total latency is 136μs and
it is approximately 35.2% of RTT
The receiving side (eth→tap) is
heavier than the sending
side (tap→eth)
14 Copyright 2018 FUJITSU LABORATORIES LIMITED
Latency Mean [μs] 50%tile [μs] 99%tile [μs]
RTT of tap on our cloud 386 377 871
Total latency on our cloud 136 124 461
Total latency on local 26 26 46
Receiving
side
Data sending (tap→eth) 24 22 54
Data receiving (eth→tap) 44 44 99
ACK sending (tap→eth) 18 17 42
ACK receiving (eth→tap) 50 41 266
16. OVS latency versus Linux bridge latency
Mean latency at each software component
Regardless of the ACK and data path, the latency at the receiving
is higher than that at the sending
OVS is lower than Linux bridge (iptables)
•Flow-cache (mega flow) on OVS would be a cause of low latency
in comparison with Linux bridge
15 Copyright 2018 FUJITSU LABORATORIES LIMITED
17. Burstiness of latency – (1)
What is the burstiness of latency in this research?
The abrupt fluctuation of latency can occur due to software
processing
RTT and latency behavior
Similar behavior is observed between RTT and total latency
16 Copyright 2018 FUJITSU LABORATORIES LIMITED
Latency[s]RTT[s]
Elapsed time [s]
18. Burstiness of latency – (2)
The quantitative characteristics of burstiness of latency
Apply the burst detection algorithm by Kleinberg (SIGKDD’02)
Detect the 856 burstiness events
Mark 10% of burstiness on the RTT
17 Copyright 2018 FUJITSU LABORATORIES LIMITED
<RTT and 10% of burstiness (a red bar corresponds to the burstiness)>
Mean Top 10% Greatest
Burst period [ms] 1.9 5.5 25
The burstiness of latency can be
a major cause of the increased RTT
19. Summary & consideration
Summary in our measurement
The total latency was increased by the receiving side regardless
of data and ACK paths
The total latency is approximately 35.2% of RTT
10% of burstiness mainly contributes to the increased RTT
Consideration
The number of CPUs for packet match/action on the host
The number of entries at iptables
The performance of Vhost thread/vswitch
18 Copyright 2018 FUJITSU LABORATORIES LIMITED
21. Causes for consideration – (2)
20 Copyright 2018 FUJITSU LABORATORIES LIMITED
This limitation would not be the major cause
22. Conclusion
Analyze the impact of software-based virtual network on
latency on the public cloud based on OpenStack
There was no assumption about the cloud
•The packet traces on the host side are directly used
The total latency was increased by the receiving side regardless
of data and ACK paths
The characteristics of latency and their burstiness were
quantitatively clarified
The latency is approximately 35.2% of RTT and 10% of
burstiness mainly contributes to the increased RTT
21 Copyright 2018 FUJITSU LABORATORIES LIMITED
23. Future work
Investigate the impact of virtual network on
different environment and architecture
Make clear a major cause of the increased
latency on the software-based virtual network
22 Copyright 2018 FUJITSU LABORATORIES LIMITED
24.
25. TCP Throughput with 1024 bytes
Throughput at eth(NIC)
Mean throughput is 4389 [Mbps]
Major traffic is the measurement
There is a small fraction of
background traffic
•The impact of background
would be small to the measurement
24 Copyright 2018 FUJITSU LABORATORIES LIMITED
26. Network applications on target cloud
How to use the public cloud?
Targets are enterprise customers and their services
•e-mail, testbed, database, storage, Web services, and development
environments
Major network applications
99% of traffic was TCP
X11 protocol was occupied 84% of traffic
25 Copyright 2018 FUJITSU LABORATORIES LIMITED
27. CDF of latency
CDF curves
The latency when the packets are received is larger than that of
sending side
26 Copyright 2018 FUJITSU LABORATORIES LIMITED
Sending
side
Receiving
side
RTT
The similar characteristics is observed