Sneak preview of why you want to use MidoNet to realize virtual router in a distributed way. Japanese is only on the Cover Page. The rest of the content is in English, so English reader, please just skip the front page.

1. 2012.12.21
CloudStack + MidoNet
分散アーキテクチャに基づいた Virtual Router
がスケーラブルな仮想ネットワークを実現する
今⽇日はマヤ暦では地球最後の⽇日ですが、ミドクラは愚直に CloudStack に MidoNet をイ
ンテグレーションのラストスパートをかけています。 今⽇日はそのスニーク・プレビュー
を⼀一⾜足早い Xmas Present としてご紹介します！
MidoNet の特徴は、その分散アーキテクチャにあります。DHCP, Source NAT など
Router の持つ機能を特定のホストや VM 上で動かすのではなく、システム全体で⾃自⽴立立分
散的に実現します。各ホスト上にある MidoNet Agent がゲスト VM と直接やり取りする
ことでトラフィックをコントロールし、CloudStack にエッジ・オーバーレイ型の仮想ネ
ットワークをインテグレーションします。
エッジに⼊入ってきたトラフィックは、構内のネットワークに送出される前に MidoNet で
演算を⾏行行い(Ingress 処理)、パケットへの必要な改変、マッチング、アクションが⾏行行われ
た後、IP トンネルと通して⽬目的となるエッジに 1 ホップで届けられます。我々はこれを
Intelligence at the edge と呼びます。必要な処理を Ingress Point で⾏行行うことでより効
率のよい、フレキシブルでスケールし易いネットワーク環境を実現します。
以下に CloudStack に MidoNet を組み合わせた時の利点や、その仕組み等をご紹介しま
す。
MidoNet に関するご質問等は、お気軽に下記メール頂くか、もしくは弊社の Facebook
Page に投稿お願いします！
Email : info@midokura.com
Facebook Page : http://www.facebook.com/MidokuraJapan/

2. How to replace CloudStackʼ’s networking functionality
This article explains why one might want to replace or extend CloudStackʼ’s existing
networking technology. We also go into moderate technical detail on how to do so,
using the CloudStack integration of Midokuraʼ’s MidoNet [1] network virtualization
technology as a case study.
Why replace CloudStackʼ’s networking functionality?
Out of the box, CloudStack provides most of its services (such as DHCP, Source NAT)
via the CloudStack Virtual Router.
The CloudStack Virtual Router is an actual VM; at the time of writing (Dec.21, 2012), it
consists of a Debian image with haproxy, dnsmasq etc installed. The number of these
Virtual Router VMs launched depends on the type of Guest Network used. For example,
when using Advanced networking and Isolated Guest Networks, one Virtual Router VM
will be launched per Guest Network. In the VPC mode, the Virtual Router is also
responsible for routing traffic among the isolated networks within the VPC.
The CloudStack Virtual Router has several disadvantages; one example is inefficient
routing of traffic in the VPC mode. Although there may be a large number of VMs
running on multiple hosts, there is only one Virtual Router VM responsible for the entire
VPC traffic, running on one host. Thus, to route among isolated networks, all traffic
must pass through this host and through the Virtual Router VM. Letʼ’s take an example
(Figure 1):
Figure 1: Traffic flow when using CloudStack Virtual Router

3. In the diagram above, if Guest VM 1 on Network 1 wants to send to Guest VM 2 of
Network 2, the traffic will first go from Host A to Host B and into the Virtual Router VM,
before being directed to Host C. This makes Host B and the Virtual Router VM a
bottleneck, and for a heavy traffic environment, this is not a scalable solution.
Ideally, the traffic in the example above should go directly from VM 1 on Host A to VM
2 on Host C, with no reliance on the Virtual Router VM. This is more clearly illustrated
when you consider a logical topology of the network. By removing the hosts from the
equation, we can see that the VMs are connected in the logical network as shown in
Figure 2. Note that we have removed the notion of Virtual Router VM and replaced it
with a logical construct called Logical Router, which is a virtual entity that you can
apply configurations such as routing and ACL to, and it is not a separate VM.
Figure 2: Example logical topology
If the hypervisor host sending traffic is aware of the logical topology shown above,
before a packet is sent out into the physical wire, it can determine the host IP of the
destination VM, and create an IP tunnel to send the packet directly to that host. This
effectively removes any reliance on the Virtual Router VM and avoids potential traffic
bottleneck.
At an extremely high level, this is exactly what MidoNet, a virtual networking platform,
does. Through the MidoNet REST API, users can construct logical topologies which are
stored in a distributed, fault-tolerant database, and the MidoNet agent running on each
host can manage tunnels and forward packets to other MidoNet hosts. To plug the
VMs into MidoNet networks, however, it must be integrated with CloudStack. Once fully
integrated, the traffic from VM 1 to VM 2 would flow as shown in Figure 3.

4. Figure 3: Traffic flow using MidoNet instead of CloudStack Virtual Router
Enabling this kind of virtualized networking to the cloud platform was one of the main
motivations for extending CloudStackʼ’s networking functionality. In the rest of this
article, we will focus on the details of the actual CloudStack-MidoNet integration work
currently in progress.
Creating a NetworkElement
Network Elements represent CloudStack network components which provide network
services (e.g. Source NAT, DHCP).
In our case, the MidoNetElement represents MidoNet, and takes care of communicating
with the MidoNet API in order to set up the virtual topology and provide various
network services like Source NAT and DHCP. Most of the logic of our integration is
implemented here.
To create your own Element, you need to extend com.cloud.network.element.
NetworkElement. Below, we give an overview of the methods the Element must
implement, an idea of what each does, and we discuss what the MidoNetElement does
as a concrete example.

5. getCapabilities()
When you create a network offering, you choose the capabilities you want your
network to have (e.g. Source NAT, DHCP), and you then choose the Provider for
each of these capabilities. If and only if a capability is listed by your Elementʼ’s
getCapabilities() call, your Element will show up in the dropdown list for Providers of
that Capability. See screenshot below where we choose MidoNet as the provider for
DHCP services in a new NetworkOffering (Figure 4).
Figure 4: Screenshot of choosing Capabilities and Providers in a new NetworkOffering
implement()
Called when the first VM is launched on a guest network which uses this Element. If
using CloudStackʼ’s Virtual Router, the Virtual Router VM would be launched at this
time. In our integration, we add a MidoNet virtual router to the MidoNet virtual
topology at this point, ready for VMs to use when they launch.
prepare()
Called when a new NIC is added to the guest network, usually meaning just before a
VM is launched. In our integration, we add the NIC to the MidoNet virtual topology,
for example setting the NIC to be connected to the guest networkʼ’s MidoNet virtual

6. router. This means when the VM comes up, its NIC will already be correctly
connected into the MidoNet topology.
release()
Called when a NIC is removed from the guest network, usually meaning just after a
VM is shut down. In our integration, we remove the NIC from the MidoNet virtual
topology, the virtual topology equivalent of plugging the network cable out of the
router.
Creating a GuestNetworkGuru
When a new Guest Network is created in CloudStack (e.g. via the UI or API),
CloudStack needs to decide which Guru to use for that network. It looks at all of the
Gurus defined in components.xml [2], and tries to load each one.
Each Guru has a canHandle() method which inspects the attributes of the Guest
Network being created, and returns true if it can handle that network, false if it canʼ’t.
When creating your own Guru, you need to keep in mind two things:
● Ensure your canHandle() method returns true only for Guest Networks you can
handle and not those you canʼ’t
○ For example, the ExternalGuestNetworkGuru returns true if the network is
Advanced and Isolated, false otherwise
● Ensure your Guru doesnʼ’t clash with another
○ If there is another Guru which can handle the same types of networks as
your Guru, two guest networks will be created in CloudStack when you try
to create one
○ Extending CloudStack Networks[2] recommends that you remove the
conflicting Guru from components.xml when deploying to avoid this issue
To create your own Guru, you need to extend
com.cloud.network.guru.GuestNetworkGuru. Below, we give an overview of the
callbacks the Guru must implement, and an idea of what each does. In our case, there
was very little we needed to do in the Guru, as the Element class takes care of most of
the logic.

7. design()
Called when the guest network is created, but before any VMs exist on the network.
The parent class (GuestNetworkGuru) does preparation such as setting the gateway
IP based on the CIDR specified by the user.
implement()
Called when the first virtual machine is about to be started on the guest network.
reserve()
Called when a new virtual machine is starting in the network.
release()
Called when a virtual machine is stopped in the network (release its allocated IP etc).
shutdown()
Called when all virtual machines within the network have been stopped, and the
guest network is garbage collected.
trash()
Called when a guest network is being deleted.
Getting CloudStack to load your classes
First, make sure the network classes you have created are registered in
components.xml.
For example, under
<adapters key="com.cloud.network.guru.NetworkGuru">
We added:
<adapter name="MidoNetGuestNetworkGuru"
class="com.cloud.network.guru.MidoNetGuestNetworkGuru"/>
and under
<adapters key="com.cloud.network.element.NetworkElement">
we added our Provider, MidoNet:

8. <adapter name="MidoNet"
class="com.cloud.network.element.MidoNetElement"/>
Restarting the management server will cause CloudStack to load these classes, and
you can now create a NetworkOffering from the UI which uses the Capabilities (e.g.
Source NAT, DHCP) provided by your Provider.
Creating a NetworkOffering from the UI
In the UI, go to Service Offerings and choose Network Offerings from the dropdown.
Click Add New Network Offering, and under Supported Services, choose your Provider
for any of the Capabilities you wish to use in the Offering, then save the Offering.
Create a Guest Network using your NetworkOffering
When creating a Guest Network in the UI, the NetworkOffering can be chosen from a
dropdown menu. However, there are a few hurdles to watch out for.
First, your Provider (e.g. MidoNet) must be enabled on the underlying Physical Network,
or else your NetworkOffering wonʼ’t appear in the dropdown list.
You can enable your provider on the physical network using the CloudStack API. For
example, to enable the MidoNet Provider using the CloudStack command line tool
CloudMonkey [3], you would use a sequence of commands like this:
cloudmonkey add networkserviceprovider name=MidoNet
physicalnetworkid=$PHYSICALNETWORKID
cloudmonkey update networkserviceprovider state=Enabled
id=$MIDONETSERVICEPROVIDERID
It is also possible to add your Provider to the CloudStack UI - this then allows you to
toggle your Provider on and off in Infrastructure > Zones > $ZONENAME > Physical
Networks > $PHYSICALNETWORKNAME > NetworkProviders instead of enabling it via
the API. We wonʼ’t go into this part in detail here, but if you search for the Providers
which are currently shown in the UI in ui/scripts/system.js, youʼ’ll quickly see which
parts you need to add in that file to display your Provider.

9. Another point to check at this stage is to ensure that the IsolationMethod of the
underlying physical network matches what you specified in the canHandle() method of
your GuestNetworkGuru.
Once all of the above is in place, your NetworkOffering should show up in the
dropdown list for Create Guest Network, and you should be able to successfully create
a network which uses your new Provider.
Plug in the NIC
The final step is to plug the VMs into the MidoNet network for connectivity.
The method for implementing this in CloudStack differs depending on the hypervisor
being used. For the first step of our integration, we are using the KVM hypervisor. For
the KVM case, we override the existing CloudStack VIFDriver so that in the plug method,
we:
● Create a TAP interface (NIC) for the new VM to use
● Plug the TAP interface (NIC) into the MidoNet datapath, so that traffic to and from
the VM flows through the MidoNet datapath
Conclusion
We hope that this overview gave you a concrete example of how to extend CloudStack
networking, and an idea of what we are aiming to achieve with our MidoNet integration
in CloudStack. We think that the scalability and flexibility of MidoNetʼ’s virtual networking
can be a great addition to CloudStack networking.
For further information on extending CloudStack networking, see [4].
References
[1] Midokura MidoNet:
http://www.midokura.com/midonet/
[2] Extending CloudStack Networking (components.xml section):

10. http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Extendin
g_CloudStack_Networking#Loading_the_components_into_CloudStack
[3] CloudStack CloudMonkey CLI:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+CloudMonke
y+CLI
[4] Extending CloudStack Networking:
http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Extendin
g_CloudStack_Networking