Openstack Networking Internals - first part
Description of the Virtual Network Infrastructure inside an OpenStack cluster
The pictures of the VNI were taken with the "Show my network state" tool
https://sites.google.com/site/showmynetworkstate/
5. ● OpenStack is a cloud platform management software
● A cloud platform is a cluster of machines that host
some servers (instances): the servers are offered to the
users as as a “service”. The user is able to create a
“virtual infrastructure” composed of servers and
network appliances (firewalls, routers, ...)
● The servers can be implemented as:
– VM (KVM,VMWare,..)
– light container (LXC-Docker,..)
– bare metal (PXE boot,…)
6. OS is composed of the following components:
● Web dashboard (Horizon)
● Compute (Nova): it manages instances lifecycle
● Keystone: credentials, service catalog of all the OS services (list of REST service
endpoints)
● Glance: image management. An image is a blob file containing a file system with a
“pre-cooked” VM: it can be used by hypervisors to boot a new instance!
● Networking(Neutron): network management
● Block storage (Cinder): persistent storage (volumes)
● Other services:
– Object storage (Swift): distributed storage for non-structured data
7. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
8. ● Data net
● Mgmt net
● External/API net
These networks are implemented as physical
separated networks.
Cesena cluster: I configured a switch with “port-based,
access mode” VLANs
9. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
Allows the admin to access the cluster nodes and
it is used for inter-service communication
EVERY NODE IS ON THIS NET
10. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
Used for inter-VM communication.
Depending on the chosen network virtualization
mechanism, the packets will be
VLAN tagged packets, or
encapsulated packets (VXLAN, GRE)
11. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
It allows the VMs to access the Internet,
the user to access the VMs
13. External net
●Keystone
●Nova
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
●API: REST endpoint, receives user requests
●Scheduler: chooses a compute node
●Glance:
●API
●Registry
●Neutron
●Server: REST API endpoint
●plugin: implements the VNI
●Cinder:
●API,…
●message queue: middleware for inter-service
communication
14. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
Neutron
• plugin: implements the VNI
• L3: virtual routers creation
• dhcp
• metadata
15. External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
Management net
Data net (flat)
Internet
In each Compute node:
●Neutron
●plugin: implements VNI
●Nova
●compute: manages the hypervisor
16. ● User send REST API calls to the service enpoints,
using:
– the web dashboard
– CLI clients
● OS components communicate between each other
using:
– Message passing (an AMQP server resides in the
controller node)
– rest API calls
● Some components (neutron-server, keystone, etc.)
access directly to a DB to save/modify their state
17. Sometimes the external network is not
directly connected the internet, but there is a
datacenter gateway allowing the access to the
internet
Public net
gateway
Internet
REST API
External net
Private
cloud
18. Public net
gateway
External net
Controller CPU node 1 CPU node 2 CPU node 3
Mgmt net
data net
nova api
Network node
REST API
User
port forwarding
Internet
REST protocol
19. Public net
gateway
External net
Internet
REST API
User
port forwarding
Controller Network node CPU node 1 CPU node 2 CPU node 3
Horizon
(Web server
apache2)
HTTP request
Mgmt net
data net
21. Public net
gateway
External net
Controller Network node CPU node 1 CPU node 2 CPU node 3
rete dati (flat)
REST API
User
port forwarding
Internet
xvncproxy
VNC protocol
Port forwarding on 6080 port!!
hypervisor
The hypervisor pipes the VM
video output through the network
sudo iptables -t nat -I PREROUTING
-i eth1 -p tcp --dport 80 -j DNAT
--to-destination 10.250.0.1:80
VM
Mgmt net
23. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
nova
compute
nova api
message
queue server
Mgmt net
data net
24. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
nova
compute
Glance API
REST endpoint
REST API call
Mgmt net
data net
25. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
Mgmt net
data net
VM VM
26. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
Mgmt net
data net
Virtual router VM VM
27. Public net
gateway
External net
Internet
REST API
NAT/
port forwarding
Controller Network node CPU node 1 CPU node 2 CPU node 3
Mgmt net
data net
Virtual router VM
User
28. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
Mgmt net
data net
VM
Virtual router
(NAT/
Floating IP)
NAT/
port forwarding
User
The VM has:
• a fixed private IP on a
private tenant net
• an optional floating IP on
the external network
The VR does:
• The NAT for
private IPs
• DNAT of the
floating IPs
29. Public net
gateway
External net
Internet
REST API
Controller Network node CPU node 1 CPU node 2 CPU node 3
data net
Mgmt net
VM
hypervisor
storage
The VM sees the
disk (block device)
as a local device!
Network file system protocol (NFS, iSCSI, …)
Storage node
31. Neutron defines these network abstractions:
Network – an isolated L2 network segment
Subnet – an IP address block on a certain network
Router – a gateway between subnets
Fixed IP – an IP on a tenant network
Floating IP – a mapping between IP of external
networks and a private fixed IP
Port – an attachment point to a network.
Users only see these abstractions!!!
Neutron implements these abstraction inside the VNI
in the cluster nodes (i.e. at the host level) so that VMs
(guest level) can see the virtual networks
32. Tenant network: a network created by a cloud
user. The VM takes a fixed IP from this net (not
modifiable after the VM creation)
◦ implementation detail: the VM receives the IP from a
DHCP server configured to give always the same fixed IP
to that VM!
Provider network: network external to the cluster,
allows outside connectivity, passing through the
network node. A VM can allocate a floating IP to
gain external visibility (OpenStack maps each
floating IP to the related fixed IP). Floating Ips are
deallocable.
34. They are “leaky abstractions”! For example:
◦ Net creation is limited to the effective VLAN ID
availability for the physical network!
◦ During the network creation you could specify low
level implementation details ( such as the VLAN ID)
or leave OpenStack decide them for you
35. server (REST API endpoint): receives API requests, saves all
the network info in a database, instructs the agents
plugin agent: implements the VNI inside the cluster node,
using the technology specified (GRE tunnel, VLAN, VXLAN, …).
dhcp: implements the dhcp servers
L3: implements the virtual routers
Metadata: the VMs contact the metadata service at the
creation
37. ● Create a L2 network
– neutron net-create net1
● Associate a L3 subnet to the network
– neutron subnet-create net1 10.0.0.0/24 --name subnet1
● Boot a VM on that subnet
– nova boot --image img --flavor flv --nic net-id=uuid vm_name
38. A user that wants to create a VM:
1. Sends a REST API (via CLI client or web dashboartd) to the Keystone REST
endpoint (request+auth) and Nova endpoint
2. “nova-scheduler”, internally, chooses the best suitable compute (CPU) node that
will host the VM
3. In that CPU node, the “nova-compute” component does the following things:
1. prepares the hypervisor
2. asks Glance the VM image
3. asks Neutron components to allocate the VNI
4. asks Cinder to allocate the persistent block storage (volumes) for the VM
40. Network namespaces is a technology that allows
to separate/isolate multiple network domains inside a single
host by replicating the network software stack
A process executed in a namespace sees only specific:
◦ physical/virtual network interfaces
◦ routing/arp tables
◦ firewall/NAT rules
You can:
◦ Create a netns
◦ Create a process/virtual network component inside that netns
41. Network
namespace 1
process
process
Linux kernel
global namespace
(PID 1)
Hardware
Network
namespace N
Network
namespace 2
...
A global namespace created
at boot time. Processes
usually reside here
You can:
• Create a netns
• create a vif inside the netns
• start a linux process
VIF
process process
42. VIF VIF
Network
namespace 1
Linux kernel
global namespace
(PID 1)
Hardware
Network
namespace N
Network
namespace 2
...
Namespaces guarantee L3
isolation, so the interfaces can
have overlapping IP addresses!
43. The virtual bridges can be connected to (physical or virtual) interfaces that
reside in different namespaces: the virtual bridges act as bridges between
the namespaces
Physical host
Network
namespace 1
Network
namespace 2
VIF packet
PIF
Virtual bridge
process
46. In each node there is:
◦ An integration bridge
◦ A bridge for each physical network, connected to
The integration bridge
The physical network interface (pif)
47. public net
gateway
External net
Mgmt net
Controller Network node CPU node 1
Data net
br-int
br-data
VM
linux
bridge
br-int
br-data
br-ex
br-int
br-data
br-data: connected to
the data net
Internet
48. Internet
gateway
External net
Controller Network node CPU node 1
br-int
br-data
VM
linux
bridge
br-int
br-data
br-ex
br-int
br-data
br-int: intermediate
bridge, act as a hub of a
star network
Mgmt net
Data net
public net
49. Internet
gateway
Controller Network node CPU node 1
br-int
br-data
VM
linux
bridge
br-int
br-data
br-ex
br-int
br-data
br-ex: connected to the
external network (only
present only in the
network node!!)
External net
Mgmt net
Data net
public net
50. Advantages: namespaces allow to manage multiple L3
functions at the host level in the same node.
This is a key enabler is implementing VNIs with advanced
functionalities: the cloud users to create overlapping virtual
L3 networks!!!
◦ Two tenants can create isolated L3 networks with the same IP addresses
The namespaces are used only in the network node (because L3
functionalities at the host level are present only inside the network node)
A new network namespace is created by OpenStack when you create…
◦ … a new virtual L2 network
◦ … a new virtual router
51. In the compute node:
◦ nova-compute makes a REST API call to neutron-server
asking for a port allocation (and a fixed IP for
the VM)
◦ neutron-agent configures the virtual bridges (br-data,
br-int) via OpenFlow protocol
◦ nova-compute boots the VM
52. In the network node:
◦ neutron-dhcp
creates a netns ("qdhcp-…")
creates a vif inside that netns
spawns a dnsmasq process (DHCP server) using that vif
◦ neutron-l3
creates a netns for the virtual router ("qrouter-…")
creates a vif inside that netns
configure the routing tables inside that netns
53. In the configuration files Neutron associates
each “physical network” to a virtual bridge.
For example:
◦ Physnet1: br-data
◦ Extphysnet: br-ex
The admin, during the creation of a provider
network, must specify the associated physical
network
54. Use case: a user
◦ creates a private user network
( 10.0.0.0/24)
◦ boots a VM on this network
source ~/devstack/openrc demo demo pass && nova
boot --key-name hc01 --image cirros-0.3.1-x86_64-
disk --flavor 2 --nic net-id=61821a27-69b8-
43c2-afa8-633304d8be50,v4-fixed-ip=10.0.0.66
myserver
55. The admin can even specify the VLAN Id used on the physical data
network (even outside the VLAN ID pool specified in the configuration
file)
The user cloud is not allowed to do this (OpenStack autonomously picks
a VLAN from the VLAN pool available in the data network)
source ~/devstack/openrc admin admin pass && neutron net-create net2 --provider:network_type
vlan --provider:physical_network physnet1 --provider:segmentation_id 1000
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 61821a27-69b8-43c2-afa8-633304d8be50 |
| name | private |
| provider:network_type | vlan |
| provider:physical_network | physnet1 |
| provider:segmentation_id | 1000 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | bbc97757-f297-4c7b-b032-e70768fe8485 |
| tenant_id | a370af83e43a432abb3adfbf976d1cf8 |
+---------------------------+--------------------------------------+
NB this VLAN is that used on the
physical network!
56. PIF on the data
network
Veth pair: a pair of
vif that act as a
pipe (everything
entering from one
exits from the
other)
57. VM
eth0
LB managment interface
Tap interface (host-level
view of the VM
interface)
Veth pair che
collega br-int al
linux bridge
interfaces
connecting the
bridges
61. Provider network (external)
NB it’s the physical external network of
the cluster
User network (internal)
The VMs will receive a fixed IP on this
net.