BGP is a popular routing protocol used in the Data Center (DC). But as the protocol that powers the Internet, it also comes armed with a lot of sophistication that scares many who think a CCIE or CCNA is required to even understand it. Watch this presentation and learn: *How BGP fits in the DC with specific use cases *How to configure and manage BGP traditionally and via new methods

1. Operationalizing BGP in the Data Center
Dinesh G Dutt, Chief Scientist
Ivan Pepelnjak, Network Architect, ipspace.net
9 February 2016

2. Introduction To The Series
§Demystify networking: make networking easy to
understand and manage
§ Data Center focused, technical
§ Include cutting edge topics like container networking and
more mundane, but relevant topics
§ Include customer case studies, demos
§ Monthly
2.9.2016 2Cumulus Networks

3. Je m’appelle Dinesh Dutt (@ddcumulus)
2.9.2016 3
§Chief Scientist at Cumulus
§Ex-Cisco Fellow
§A key architect of many of Cisco’s products from Cat6k
to MDS to Nexus family of switches, including many
Cisco initiatives
§Co-author of VxLAN and TRILL drafts
§Filed for over 40 patents
Cumulus Networks

4. 4 © ipSpace.net 2016 BGP in Data Center
Who is Ivan Pepelnjak (@ioshints)
Past
• Kernel programmer, network OS and web developer
• Sysadmin, database admin, network engineer, CCIE
• Trainer, course developer, curriculum architect
• Team lead, CTO, business owner
Present
• Network architect, consultant, blogger, webinar and book author
Focus
• SDN and network automation
• Large-scale data centers, clouds and network virtualization
• Scalable application design
• Core IP routing/MPLS, IPv6, VPN
More @ ipSpace.net/About

5. Agenda
§Why BGP
§Configuring BGP
§Troubleshooting BGP
§BGP Lifecycle Management
§Customer Use Cases
2.9.2016 5Cumulus Networks

6. Key Takeaways
§BGP is an excellent fit as the routing protocol for
the DC
§BGP configuration and troubleshooting can be
simplified a lot
§Use of Unnumbered interfaces in the DC provides
tremendous benefits
2.9.2016 6Cumulus Networks

7. •Routing Protocol Architecture
2.9.2016 7Cumulus Networks
Spine
Leaf
L3 Fabric

8. 8 © ipSpace.net 2016 BGP in Data Center
Selecting a Routing Protocol in Layer-3 Leaf-and-Spine Fabric
The choices:
• OSPF
• IS-IS
• BGP
Why BGP?
• Simple
• Scalable and mature
• Supports IPv4 and IPv6
Leafs
Spine
Redundantly connected servers
?
The usual (false) objections
• BGP is complex
• BGP is hard to understand and troubleshoot
• BGP configuration is a mess

9. •Configuring BGP
2.9.2016 9Cumulus Networks
Spine
Leaf
L3 Fabric

10. Basics of BGP Configuration in DC
§ Based on draft-ietf-rtgwg-bgp-routing-large-dc-07
§ Use EBGP
§ To avoid BGP path hunting in CLOS networks:
§ All spines have the same ASN
§ Each leaf has a different ASN
§ What ASN ?
§ 2B or 4B ASN
§ Public or Private ASN
§ Cumulus supports all choices
§ Strip ASNs when advertising paths outside data center
2.9.2016 10

11. Traditional BGP Configuration
2.9.2016 11
router bgp 64501
bgp log-neighbor-changes
bgp router-id 10.0.0.1
!
neighbor 10.1.1.1 remote-as 65000
neighbor 10.1.2.1 remote-as 65000
neighbor 10.1.3.1 remote-as 65000
neighbor 10.1.4.1 remote-as 65000
router bgp 64502
bgp log-neighbor-changes
bgp router-id 10.0.0.2
!
neighbor 10.1.1.3 remote-as 65000
neighbor 10.1.2.3 remote-as 65000
neighbor 10.1.3.3 remote-as 65000
neighbor 10.1.4.3 remote-as 65000
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.17
!
neighbor 10.1.1.0 remote-as 64501
neighbor 10.1.1.2 remote-as 64502
…
neighbor 10.1.1.32 remote-as 64517
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.20
!
neighbor 10.1.4.0 remote-as 64501
neighbor 10.1.4.2 remote-as 64502
…
neighbor 10.1.4.32 remote-as 65534
router bgp 64516
bgp log-neighbor-changes
bgp router-id 10.0.0.16
!
neighbor 10.1.1.33 remote-as 65000
neighbor 10.1.2.33 remote-as 65000
neighbor 10.1.3.33 remote-as 65000
neighbor 10.1.4.33 remote-as 65000
L1 L2 L16
S1
S4
Cumulus Networks
SPINE
LEAF L1 L2 L16
S1 S2 S4S3
10.1.1.1
10.1.1.0
10.1.4.33
10.1.4.32

12. 12 © ipSpace.net 2016 BGP in Data Center
Challenge: Addressing on Core (Leaf-to-Spine) Links
Why not?
• Simplifies configurations
• No IPAM needed for the fabric
• Plug-and-play networking
How?
• Turn core subnets into P2P links
• Use unnumbered interfaces
Leafs
Spine
Redundantly connected servers
Do we need IP addresses
on core links?

13. BGP Unnumbered Configuration
2.9.2016 13
router bgp 64501
bgp log-neighbor-changes
bgp router-id 10.0.0.1
!
neighbor swp1 remote-as 65000
neighbor swp2 remote-as 65000
neighbor swp3 remote-as 65000
neighbor swp4 remote-as 65000
router bgp 64502
bgp log-neighbor-changes
bgp router-id 10.0.0.2
!
neighbor swp1 remote-as 65000
neighbor swp2 remote-as 65000
neighbor swp3 remote-as 65000
neighbor swp4 remote-as 65000
router bgp 64516
bgp log-neighbor-changes
bgp router-id 10.0.0.16
!
neighbor swp1 remote-as 65000
neighbor swp2 remote-as 65000
neighbor swp3 remote-as 65000
neighbor swp4 remote-as 65000
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.17
!
neighbor swp1 remote-as 64501
neighbor swp2 remote-as 64502
…
neighbor swp16 remote-as 64516
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.20
!
neighbor swp1 remote-as 64501
neighbor swp2 remote-as 64502
…
neighbor swp16 remote-as 64516
L1 L2 L16
S1
S4
Cumulus Networks
SPINE
LEAF L1 L2 L16
S1 S2 S4S3

14. Removing the Need For Specifying Specific Remote AS
2.9.2016 14
router bgp 64501
bgp log-neighbor-changes
bgp router-id 10.0.0.1
!
neighbor swp1 remote-as external
neighbor swp2 remote-as external
neighbor swp3 remote-as external
neighbor swp4 remote-as external
router bgp 64502
bgp log-neighbor-changes
bgp router-id 10.0.0.2
!
neighbor swp1 remote-as external
neighbor swp2 remote-as external
neighbor swp3 remote-as external
neighbor swp4 remote-as external
router bgp 64516
bgp log-neighbor-changes
bgp router-id 10.0.0.16
!
neighbor swp1 remote-as external
neighbor swp2 remote-as external
neighbor swp3 remote-as external
neighbor swp4 remote-as external
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.17
!
neighbor swp1 remote-as external
neighbor swp2 remote-as external
…
neighbor swp16 remote-as external
router bgp 65000
bgp log-neighbor-changes
bgp router-id 10.0.0.20
!
neighbor swp1 remote-as external
neighbor swp2 remote-as external
…
neighbor swp16 remote-as external
L1 L2 L16
S1
S4
Cumulus Networks
SPINE
LEAF L1 L2 L16
S1 S2 S4S3

15. Savings In IPv4 Address Utilization
Spine Leaf Total
Cumulus 4 16 20
Traditional BGP 4+ 4*16= 68 16+ 16*4= 80 148
Cumulus Networks2.9.2016 15
Spine Leaf Total
Cumulus 16 96 112
Traditional BGP 16 + 16*96 = 1552 96 + 96*16 = 1632 3184
Case 1
Case 2

16. Automation Benefit: # Variables Used in Playbook
Spine Leaf Total
Cumulus 1 + 1 (loopback subnet + spineASN) 1 (Leaf ASN base, same loopback
subnet)
3
Traditional
BGP
4+(4*16)+1 = 69
(Router IDs +
Total switches*TORIPv4 +
ASN)
16+(16*4) +16 = 96
(Router IDs +
Total switches*uplink IPv4 +
ASN)
165
Cumulus Networks2.9.2016 16
Spine Leaf Total
Cumulus 1 + 1 (loopback subnet + spineASN) 1 (Leaf ASN base, same loopback
subnet)
3
Traditional
BGP
16+(16*96)+1 = 1552
(Router IDs +
Total switches*TORIPv4 +
ASN)
96+(96*16) +96 = 1728
(Router IDs +
Total switches*uplink IPv4 +
ASN)
3280
Case 1
Case 2

17. How BGP Unnumbered Works
§Not really unnumbered: Uses IPv6 Link local
address for BGP Sessions
§Uses IPv6 Router Advertisement to learn
neighbor’s link local address
§Uses RFC 5549 to support advertising IPv4
addresses over IPv6 session
§Works on Servers and Routers
2.9.2016 17Cumulus Networks

18. 18 © ipSpace.net 2016 BGP in Data Center
The Myths of Unnumbered Interfaces
The myths of unnumbered interfaces
• Hop-by-hop Telnet è use OOB management or IPv6 LLA
• Traceroute è use RFC 5837
• Discovery tools è tell the vendor to fix them
• RSVP-TE ERO – it doesn’t apply to the data center
Leafs
Spine
Unnumbered P2P link
Other considerations
• OSPF behavior (subnet mismatch)
– OSPF Unnumbered is defined in base RFC
• EBGP sessions need IP addresses
– IPv6 Link Local Address works
• BFD over unnumbered links
– Supported over IPv6 Link Local address

19. 19 © ipSpace.net 2016 BGP in Data Center
The Myths of Unnumbered Interfaces
The myths of unnumbered interfaces
• Hop-by-hop Telnet è use OOB management or IPv6 LLA
• Traceroute è use RFC 5837
• Discovery tools è tell the vendor to fix them
• RSVP-TE ERO – it doesn’t apply to the data center
Leafs
Spine
Unnumbered P2P link
Conclusion: the benefits far outweigh the potential drawbacks
Other considerations
• OSPF behavior (subnet mismatch)
– OSPF Unnumbered is defined in base RFC
• EBGP sessions need IP addresses
– IPv6 Link Local Address works
• BFD over unnumbered links
– Supported over IPv6 Link Local address

20. Further Points About Unnumbered Interfaces
§Reduces FIB size
§Reduces attack vector since only single reachable
address
§ As opposed to as many addresses as there are links
§See RFC 7404 for more details
2.9.2016 20Cumulus Networks

21. Further Notes on Cumulus Unnumbered Support
§Traceroute works correctly
§ Not because of RFC 5837
§BFD supported over unnumbered interfaces
§PTM (Prescriptive Topology Manager) can be used
to verify cabling correctness
2.9.2016 21Cumulus Networks

22. Additional Configuration Notes
§ Setting source of routes
§ To ensure ping/traceroute work
§ Handling anycast IPs
§ Use as-path multipath-relax
§ Enable import-check
§ Timer settings
§ Advertisement-interval is 0
§ For faster convergence
§ Use peer-groups
2.9.2016 22Cumulus Networks

23. •Troubleshooting BGP
2.9.2016 23Cumulus Networks

24. Display hostname in BGP outputs
2.9.2016 24
§ IETF draft
§ draft-walton-bgp-
hostname-capability-
02
§ IANA assigned BGP
capability code
Cumulus Networks

25. Debugging BGP Options
§Support for debugging:
§ Specific prefixes
• For route-updates
• For bestpath computation
• For RIB interaction
§ Specific neighbors
• Session setup
• Keepalive
§ Other debug supports include dynamic update-groups
and Next Hop Tracking (NHT)
2.9.2016 25Cumulus Networks

26. JSON Outputs
§Common commands for BGP support JSON
outputs
§A lot more commands with JSON support coming
in 3.0
§Pro Tip: Pipe output of JSON output through python
–m json.tool for neatly formatted JSON output
2.9.2016 26Cumulus Networks

27. •Lifecycle Management
2.9.2016 27Cumulus Networks

28. Changes to Running Config
§Modifications to existing configuration are done by
updating a configuration file and reloading service
§ Only differences are applied to running service
§ Session reset doesn’t occur unless necessary
§Inbound/outbound Route-map modifications can be
tested before application
§ show ip bgp neighbor <nbr> advertised-routes route-
map <route-map-name>
§ show ip bgp neighbor <nbr> received-routes route-map
<route-map-name>
2.9.2016 28Cumulus Networks

29. Removing Node for Scheduled Maintenance
§Max-med support
§ For graceful removal of node from network
2.9.2016 29Cumulus Networks

30. Upgrading Quagga
§Quagga upgradeable as a package
§ Does not require complete binary upgrade
2.9.2016 30Cumulus Networks

31. Customer Validation
§Everything described in this presentation has been
deployed in production:
§ In mega-scale DC
§ In many medium scale DCs with 4-16 spines and 16-100
racks (ToRs)
§ In many small scale DC with 2 spines and 4-8 racks (ToRs)
§ With plain L3 workloads (such as Big Data) and with Cloud
(VxLAN overlays)
§Cumulus routing protocol suite also deployed on bare
metal servers, servers running VMs, and with
containers
2.9.2016 31

32. 32 © ipSpace.net 2016 BGP in Data Center
Summary
Why BGP?
• Simple, scalable, multi-protocol and mature
• BGP administration can be tremendously simplified
• Unnumbered interfaces make a lot of sense in the DC
Leafs
Spine
Redundantly connected servers
BGP

33. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the “Marks”) are trademarks and service marks of Cumulus Networks, Inc. in the U.S. and other
countries. You are not permitted to use the Marks without the prior written consent of Cumulus Networks. The registered trademark Linux® is used pursuant to a sublicense from
LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. All other marks are used under fair use or license from their respective owners.
§Thank You!
cumulusnetworks.com 33
Bringing the Linux Revolution to Networking
2.9.2016