Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OpenStack networking juno l3 h-a, dvr

8,146 views

Published on

OpenStack networking - juno L3 H/A & DVR

Published in: Technology
  • Login to see the comments

OpenStack networking juno l3 h-a, dvr

  1. 1. OpenStack Networking - Juno - - DVR & L3 High Availability Paul Sim Technical Account Manager paul.sim@canonical.com
  2. 2. ● Distributed Virtual Router ○ Packet flow ○ Architecture ■ SNAT, ■ DNAT(Floating IP) ■ East<->West ● L3 High Availability Index
  3. 3. DVR (Distributed Virtual Router) - Installation Network node Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent External network Compute node - 1 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3-agent Management Data Compute node - 2 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3-agent
  4. 4. DVR (Distributed Virtual Router) - Packet flow Compute node - 1 br-ex VM VM GRE Tunnel VM br-int Network node br-ex br-tun br-int br-tun Compute node - 2 VM VM VM br-int br-tun 1.SNAT External network 3. East-West traffic 2. Floating IP OVS bridge
  5. 5. DVR (Distributed Virtual Router) - SNAT : Network node Namespace OVS bridgeNetwork node qdhcp- br-tun eth0 br-int patch-tun patch-int gre~ snat- qrouter- tap taptap sg~ 50.50.6.2ns~ qr~ qg~ 192.168.10.109 SNAT br-ex tap
  6. 6. DVR (Distributed Virtual Router) - SNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge sg~on networknode packet flow
  7. 7. traffic flow DVR (Distributed Virtual Router) - SNAT : Compute node Namespace OVS bridge Linux bridge Compute node VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ sg~(50.50.6.2)on networknode ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter- 20838b7d-a7ac-4da9-92aa-adec797d600e ip rule ls 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter- 20838b7d-a7ac-4da9-92aa-adec797d600e ip route show table 842139137 default via 50.50.6.2 dev qr-9722faba-b7
  8. 8. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair
  9. 9. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9- 92aa-adec797d600e ip rule ls 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 32770: from 50.50.5.5 lookup 16 842138881: from 50.50.5.1/24 lookup 842138881 842138881: from 50.50.5.1/24 lookup 842138881 842139137: from 50.50.6.1/24 lookup 842139137 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac-4da9- 92aa-adec797d600e ip route show table 16 default via 169.254.31.29 dev rfp-20838b7d-a
  10. 10. DVR (Distributed Virtual Router) - Floating IP/DNAT : Compute node Compute node Namespace OVS bridge VM br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow br-ex tap eth0 fip- fpr~ rfp~ fg~ RouteRoute NAT veth pair ubuntu@ubuntu-6:~$ sudo ip netns exec fip-02f9d340-2caa- 4c05-86fb-460c9580f9df ip route show default via 192.168.10.1 dev fg-f3887d61-2d 192.168.10.114 via 169.254.31.28 dev fpr-20838b7d-a
  11. 11. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request ICMP Reply i.e., ping 50.50.5.3 -> 50.50.6.3
  12. 12. DVR (Distributed Virtual Router) - East-West traffic flow : network topology
  13. 13. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ VM 50.50.6.3 ubuntu@ubuntu-6:~$ sudo ip netns exec qrouter-20838b7d-a7ac- 4da9-92aa-adec797d600e ip link 2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff 5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff ubuntu@ubuntu-8:~$ sudo ip netns exec qrouter-20838b7d-a7ac- 4da9-92aa-adec797d600e ip link 2: qr-ecffa2a6-dd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:15:1e:e0 brd ff:ff:ff:ff:ff:ff 5: qr-9722faba-b7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether fa:16:3e:71:3d:5a brd ff:ff:ff:ff:ff:ff VM 50.50.5.3
  14. 14. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 SRC MAC : fa:16:3e:71:3d:5a SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3 SRC MAC : fa:16:3e:71:3d:5a SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3 SRC MAC : fa:16:3e:ce:8c:35 SRC IP : 50.50.5.3 DST MAC : fa:16:3e:15:1e:e0 DST IP : 50.50.6.3 GRE tunnel 0x3 SRC MAC : fa:16:3f:5e:a0:cf SRC IP : 50.50.5.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.6.3
  15. 15. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Reply 50.50.6.3 -> 50.50.5.3 SRC MAC : fa:16:3e:15:1e:e0 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ff:85:9b DST IP : 50.50.5.3 SRC MAC : fa:16:3e:15:1e:e0 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35 DST IP : 50.50.5.3 SRC MAC : fa:16:3e:ff:85:9b SRC IP : 50.50.6.3 DST MAC : fa:16:3e:71:3d:5a DST IP : 50.50.5.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 GRE tunnel 0x1 SRC MAC : fa:16:3f:72:60:33 SRC IP : 50.50.6.3 DST MAC : fa:16:3e:ce:8c:35 DST IP : 50.50.5.3
  16. 16. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 table=0, n_packets=9178, n_bytes=1009035, idle_age=17470, hard_age=65534, priority=1 actions=NORMAL table=0, n_packets=2066, n_bytes=214544, idle_age=5, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1) table=1, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=1,dl_vlan=2,dl_src=fa:16:3e:71:3d:5a actions=mod_dl_src:fa:16:3f:5e:a0:cf,resubmit(,2) table=2, n_packets=1849, n_bytes=183458, idle_age=5, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) table=20, n_packets=1765, n_bytes=172970, idle_age=5, hard_age=65534, priority=2,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,set_tunnel:0x3,output:3
  17. 17. DVR (Distributed Virtual Router) - East-West traffic flow : Compute node Compute node-2 Namespace OVS bridge VM 50.50.6.3 br-int br-tun gre~ qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~ Linux bridge packet flow Compute node-1 tap~ qr~ 50.50.5.1 VM 50.50.5.3 br-int br-tun qvo~ qbr~ tap~ qvb~ patch-int qrouter- qr~ 50.50.6.1 patch-tun tap~tap~ qr~ 50.50.5.1 gre~ ICMP Request 50.50.5.3 -> 50.50.6.3 Segmentation ID : 50.50.5.0/24 : 0x1 50.50.6.0/24 : 0x3 MAC 50.50.6.3 : fa:16:3e:ff:85:9b 50.50.6.1 : fa:16:3e:71:3d:5a 50.50.5.1 : fa:16:3e:15:1e:e0 50.50.5.3 : fa:16:3e:ce:8c:35 DVR Host MAC : Compute Node -1 : fa:16:3f:5e:a0:cf Compute Node -2 : fa:16:3f:72:60:33 table=0, n_packets=1789, n_bytes=175146, idle_age=17, hard_age=65534, priority=2,in_port=3,dl_src=fa:16:3f:5e:a0:cf actions=resubmit(,1) table=1, n_packets=1765, n_bytes=172970, idle_age=17, hard_age=65534, priority=4,dl_vlan=2,dl_dst=fa:16:3e:ff:85:9b actions=strip_vlan,mod_dl_src:fa:16:3e:71:3d:5a,output:8 table=0, n_packets=1857, n_bytes=184993, idle_age=18, hard_age=65534, priority=1,in_port=2 actions=resubmit(,3) table=3, n_packets=1993, n_bytes=195880, idle_age=18, hard_age=65534, priority=1,tun_id=0x3 actions=mod_vlan_vid:2,resubmit(,9) table=9, n_packets=1789, n_bytes=175146, idle_age=18, hard_age=65534, priority=1,dl_src=fa:16:3f:5e:a0:cf actions=output:1
  18. 18. L3 High Availability - Installation Network node-1 Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent External network Management Data KeepAlived Network node-2 Neutron server eth1 eth2 eth0 Neutron ML2 plugin Neutron metadata- agent Neutron L3/dhcp- agent KeepAlived Compute node - 1 Nova compute eth1 eth2 eth0 Neutron ML2 plugin Compute node - 2 eth1 eth2 eth0 Neutron ML2 plugin Nova compute
  19. 19. Network node-2 Compute node - 3 Compute node - 2 Network node-1 vRouter A - Master L3 High Availability Compute node - 1 Subnet 1 Subnet 3 Subnet 2 Subnet 5 vRouter B - Backup vRouter C - Backup vRouter D - Master vRouter C - Master vRouter D - Backup vRouter A - Backup Subnet 3 Subnet 4 vRouter B - Master Tenant X Tenant Y Tenant Z VRRP
  20. 20. L3 High Availability Namespace OVS bridge Network node-1 qdhcp- br-tun br-int qrouter- ha~ ns~ qr~qg~ br-ex Network node-2 qdhcp- br-tun br-int qrouter- qr~qg~ br-ex ns~ KeepAlived KeepAlived ha~ ubuntu@ubuntu-5:~$ sudo ip netns exec qrouter-d8625260-88a1-4312-b788-c04fc9094356 tcpdump -n -i ha-27fe59da-a8 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ha-27fe59da-a8, link-type EN10MB (Ethernet), capture size 65535 bytes 16:16:25.213440 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:27.214607 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:29.215796 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 16:16:31.216986 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20
  21. 21. L3 High Availability Network node-1 qdhcp- br-tun eth0 br-int patch-tun patch-int qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Network node-2 qdhcp- br-tun eth0 br-intpatch-tun patch-int gre~ qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Namespace OVS bridge gre~
  22. 22. L3 High Availability Network node-1 qdhcp- br-tun eth0 br-int patch-tun patch-int qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Network node-2 qdhcp- br-tun eth0 br-intpatch-tun patch-int gre~ qrouter- tap taptap ha~ns~ qr~ qg~ br-ex tap Namespace OVS bridge gre~
  23. 23. L3 High Availability Namespace OVS bridge Network node-1 qdhcp- br-tun br-int qrouter- ha~ ns~ qr~qg~ br-ex KeepAlived ubuntu@ubuntu-5:~$ cat /var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/keepalived.conf vrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/d8625260-88a1-4312-b788- c04fc9094356/notify_fault.sh" } vrrp_instance VR_1 { state BACKUP interface ha-27fe59da-a8 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-27fe59da-a8 } virtual_ipaddress { 192.168.10.118/24 dev qg-8fffbd7e-8a } virtual_ipaddress_excluded { 50.50.1.1/24 dev qr-dee474e1-1e } virtual_routes { 0.0.0.0/0 via 192.168.10.51 dev qg-8fffbd7e-8a } }
  24. 24. Network node Tenant A L3 High Availability Namespace OVS bridge br-tun br-int qrouter- ha~ br-ex KeepAlived qrouter- ha~ KeepAlived HA network : 169.254.192.0/18 Segmentation id : 0x6 Tenant B qrouter- ha~ KeepAlived qrouter- ha~ KeepAlived HA network : 169.254.192.0/18 Segmentation id : 0x7 ● One KeepAlived instance per vRouter ● One HA network per tenant ○ Each HA network has separate segmentation id ○ allow_overlapping_ips = True ● Maximum 255 HA routers per tenant.

×