These are the slides that I used in a presentation given to the Colorado OpenStack meetup group on Wednesday, September 17, 2014.

1. Distributed Virtual Router
Introduced in the Juno Release of OpenStack Neutron
Carl Baldwin
DVR Illustrations courtesy of Jack McCann
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

2. DVR Architecture
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

3. Neutron deployment architecture without DVR
neutron-server
API
auth
ML2 plug-in
database
Network Service Node(s)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 3 to change without notice.
Compute hosts
DHCP agent
L3 agent
message
queue
Metadata agent
ovs agent
Nova metadata
ovs agent
Open
vSwitch
Open
vSwitch

4. Neutron deployment architecture with DVR
neutron-server
API
auth
ML2 plug-in
database
Metadata agent
L3 agent
message
queue
Network Service Node(s)
Compute hosts
DHCP agent
Metadata agent Nova metadata
L3 agent agent_mode = dvr
ovs agent
w/dvr agent
Nova metadata
ovs agent
Open
vSwitch
Open
vSwitch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 4 to change without notice.
enable_distibuted_routing = True
programs DVR flow handling
external network
compute nodes on external
network

5. API extension
Adds ‘distributed’ attribute to ‘router’ object
• Can be set by admin user through the API
• Global default is set as “router_distributed” in neutron.conf
• Default is False
• The attribute is only visible to admin tenant in GET
• Cannot be updated
• Work in progress to allow update from False to True
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 5 to change without notice.

6. DVR – East-West (subnet-to-subnet)
“VM1-1
QRouter-X
S1.1 S2.1
br-int
patch-tun
eth0
ARP for gw
kept local
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 6 to change without notice.
“VM2-1
br-int
patch-tun
eth0
QRouter-X
S1.1 S2.1
same gw IP/MAC
on each node
no remote bcast
in to routers

7. DVR – North-South (floating IP)
“VM1-1
“VM2-1
br-int
QRouter-Y
qr rfp-y
floating IP namespace
br-ex
external-vlan
eth0
QRouter-X
qr rfp-x
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 7 to change without notice.
floating IP NAT in router ns
default route via FIP-NS
local addressing connects QR to FIP-NS
floating IP host routes pointing to QR
proxy-arp for floating IPs on br-ex
(future option: BGP route injection)
fpr-x fpr-y
fg-u

8. DVR – North-South (default SNAT)
br-int
snat-Y
qr qg-x
br-ex
external-vlan
eth0
snat-X
qr qg-x
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 8 to change without notice.
default SNAT in snat namespace
default route via br-ex

9. Database
router_extra_attributes
router_id string uuid
distributed boolean
dvr_host_macs
host string 255
mac_address string 32
ml2_dvr_port_bindings - port binding for all the
ports associated to a DVR identified by router_id
port_id string uuid
host string
router_id string uuid
vif_type string
vif_details string
vnic_type string
profile string
cap_port_filter boolean
driver string
segment string
status string
csnat_l3_agent_bindings
router_id string uuid
l3_agent_id string uuid
host_id string
csnat_gw_port_id string uuid
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 9 to change without notice.

10. config file options
neutron.conf
[DEFAULT]
router_distributed = False
dvr_base_mac = fa:16:3f:00:00:00
ovs_neutron_plugin.ini
[agent]
enable_distributed_routing = False # Make the l2 agent run in dvr mode
l3_agent.ini
[DEFAULT]
agent_mode = legacy # legacy, dvr, or dvr_snat
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 10 to change without notice.

11. OVS Flow Handling
How to Distribute the Router’s Internal Port
• ARP Requests to Router Port are Blocked from the Tunnel
• These ARP requests should only be seen by the local port
• Source Mac is Mapped to Host Mac on Overlay Network
• All traffic generated by the
• A mac address is allocated for each compute host
• Mapping must be done on both ends of the tunnel
• Destination Mac Blocked from Overlay
• These should go to the local port
• They would create mac ambiguity in the overly
• L2 Pre-Population is Required
• “Prevent(s) multiple unicast of routed packets destined to remove VMs.”
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 11 to change without notice.

12. DVR Limitations
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

13. Default SNAT still centralized
snat-Y
qr qg-x
br-ex
external-vlan
eth1
“VM1-1
qrouter-X
S1.1 S2.1
br-int
patch-tun
eth0
br-int
patch-tun
eth0
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 13 to change without notice.

14. Floating IP Namespace
• Pros
• Eliminates Need for Public Address/IR
• Keeps IR Macs Off External Net
• Cons
• Extra Complexity in L3 Agent
• Still Consumes a Public Address / CN
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 14 to change without notice.
QRouter-Y
qr rfp-y
fpr-x fpr-y
floating IP namespace
fg-u
eth0
QRouter-X
qr rfp-x
br-ex
external-vlan

15. Heavy L2 Integration
• Led to Initial Dependence on OVS and Tunnel Protocols
• Mitigate Scope Creep
• Distributed Port Concept Needs to be Abstracted
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 15 to change without notice.

16. Firewall as a Service (FWaaS) Complexity
• External Net Connects to Hypervisors
• FWaaS is Needed There Too.
• Asymmetric Routing Problem
• E/W Routing
“VM1-1
QRouter-X
S1.1 S2.1
br-int
patch-tun
eth0
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 16 to change without notice.
“VM2-1
br-int
patch-tun
eth0
QRouter-X
S1.1 S2.1

17. Contributing DVR
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

18. Initial Development on Havana
Pros and Cons
• Stable Code Base
• No Risk of External Regression
• Very Large Effort to Integrate
• Upstream Moves Quickly
• Subject to Regression
• Comm. Standard Enforcment
• Code Style
• No Demand for Unit or Functional Tests
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 18 to change without notice.

19. Initial Development on Havana
If We had to do it Over Again…
• Contribute Smaller, More Focused Patches to Trunk
• Start with Pure Refactoring Needed to Ease Development
• Develop Unit Tests for Code that will be Modified
• Move Gradually Beyond Refactoring to Other Improvements
• Divide Implementation According to Themes
• Develop Unit Tests (TDD) and Functional Tests to Prevent Regression
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 19 to change without notice.

20. Divided in to 7 Patches
• Division According to Component
• Patches Added Unused Code for Later Patches
• Indicates there are themes that cross patch boundaries
• Each Patch Had Multiple Active Authors
• Indicates possibly more than one theme in the patch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 20 to change without notice.

21. Handling Multiple Changes
• Dependency Order Not Linear
• Should it be?
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 21 to change without notice.

22. Handling Multiple Changes
It is Never Linear!!!
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 22 to change without notice.

23. Handling Multiple Changes
Enemy Number One!
• Automatic Rebase Feature
• Default behavior of “git review”
• Most of the Time it is Disruptive
• Sometimes it is Destructive
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 23 to change without notice.

24. Handling Multiple Changes
Enemy Number One!
• Clobbered API Extension!!
• Happened More than Once
• Use --no-rebase Always
• Rebase on Merge Conflict
• Work from the bottom up
• Merge Faster
• Smaller, more focused patches
• Continuous community involvement
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 24 to change without notice.