1. CATNIX summarized the migration of its peering platform over 20 years from 1999 to 2022, increasing bandwidth capacity from 256 kbps to 100 Gbps. 2. The recent migration in 2021 involved upgrading switches to increase switching capacity and port speeds while eliminating oversubscription and adding deep buffers. 3. The migration process took place in stages, including preparing new cabling, testing in a lab, migrating services equipment port-by-port, and configuring EVPN VXLAN and ECMP routing between the new switches. 4. A potential "horror story" was avoided when equipment was accidentally reset in the lab, but configuration backups saved the migration plan.

1. Migrations with Happy Destinations
Maria Isabel Gandía Carriedo
CATNIX / CSUC
EURO-IX Meeting
Edinburgh, 11-10-2022

2. Timeline
1999
2002 2008
2021
CATNIX is born
6 participants
BW > 256 kbps
1 Gbps
European
Peering Forum
10 years
22 participants
BW >10 Mbps
10 Gbps
3 sites
Route
servers
20 years
46 participants
BW >1 Gbps
100 Gbps
RIPE
meeting
2017
ESNOG meetings
Euro-IX Forums
2011 2015
DNS Root
servers
2005
2 sites
2003 2009
2016 2018
2014
2019
2007
2022
= New platform

3. Evolution of the Peering Platform
2008-2016 2016-2021 2021-
Switches Cisco Catalyst
4507 Sup6E
Arista 7280SE Arista 7504R and
7280SR
Switching capacity 320 Gbps 1.44 Tbps 38 Tbps
2.16 Tbps
Oversubscription Yes No No
Deep Buffers No Yes Yes
10 Gbps ports 40 72 ≥176
≥48
40 Gbps ports  ✓ ✓
100 Gbps ports  2 32
6

4. Old Equipment
2x10 Gbps
3x10 Gbps 10 Gbps
Arista
7280SE
Arista
7280SE
2x10 Gbps
Arista
7280SE
Cisco
C3650-48TQ
Cisco
ASR1001-X
bitNAP
Campus Nord Equinix

5. New switches and Architecture
100 + 4x10 Gbps 100 Gbps
2x100 Gbps
Arista
7280SR
Arista
7280SR
Arista
7504R
Arista
7020SR
bitNAP
Campus Nord Equinix

6. 1. Housekeeping

7. 2. New Cabling
UTP Cat.6
Patchsee
SM OS2 G652.D
MM OM4

8. 3. Lab
Arista 7504R, 7280SR & 7020SR, deep buffers, 1/10 and 100 Gbps ports
1. Testing connections,
configuration and new
architecture.
2. eVPN VxLAN + ECMP + BFD is:
1. More flexibility
2. More scalability
3. Uniform distribution of the payload
4. Opportunities to grow
5. Latency was not affected.

9. 4. Migration of the Services Equipment
Arista 7020 SR:
• 1,04 Tbps
• 3 GB Deep buffer
• 32x1/10 Gbps ports
• 2x100 Gbps
• VOQ Architecture
• No oversubscription
1. Prepare new cables without connecting them.
2. Take pictures of all the cables.
3. Take a snapshot of the status of the
interfaces, peerings & configurations.
4. Graceful BGP shutdown (RFC 8326).
5. Connect new services equipment to the
Peering switch at Campus Nord.
6. For each connection:
1. Remove transceiver from the old equipment.
2. Insert transceiver in the new interface by
category sorting (SM-MM-UTP).
3. Use new cable to connect service.
4. Remove old cable.
5. Check connection.

10. 1. Prepare the new cables without connecting them.
2. Take pictures of all the cables.
3. Take a snapshot of the status of the interfaces, peerings & configurations.
4. BGP culing – RFC8327 (sessions down, traffic not interrupted) on
interfaces between switches (BGP intra-switch up; inter-switches down).
5. Shutdown the interfaces between the old switches.
6. Turn on connections between new switches (100 Gbps / 3x10 Gbps).
7. Interconnect the old and the new switch in each node.
Equinix bitNAP
Campus Nord
Arista
7280SR
Arista
7280SR
Arista
7504R
Arista
7280SE
Arista
7280SE
Arista
7280SE
5. Migration of the Peering Platform (I)
X X

11. 5. Migration of the Peering Platform (II)
8. BGP culing – RFC8327 with the services router.
9. Upgrade services router connection to 2x100 Gbps.
10.For each physical connection (89) / port-channel (21):
1. Check interface status and BGP session with AS13041.
2. BGP culing on the interface.
3. Remove transceiver(s) from the old equipment.
4. Insert transceiver(s) in the new interface by category sorting (SM-MM-UTP).
5. Use new cable to connect member.
6. Remove old cable.
7. Check connection.
Arista
7020SR
2x100 Gbps
Equinix bitNAP
Campus Nord
Arista
7280SR
Arista
7280SR
Arista
7504R
Arista
7280SE
Arista
7280SE
Arista
7280SE
X X

12. 5. Migration of the Peering Platform: Campus Nord
Arista 7280SR
• 2,16 Tbps
• 4 GB Deep buffer
• 48x1/10 Gbps ports
• 6x100 Gbps
• VOQ Architecture
• No oversubscription

13. 5. Migration of the Peering Platform: Equinix
Arista 7504R
• 25.6 Tbps
• 4 GB / 12 GB Deep buffer
• 2 supervisors
• 48x1/10 Gbps ports
• 38x100 Gbps
• VOQ Architecture
• No oversubscription

14. 5. Migration of the Peering Platform: bitNAP
Arista 7280SR
• 2,16 Tbps
• 4 GB Deep buffer
• 48x1/10 Gbps ports
• 6x100 Gbps
• VOQ Architecture
• No oversubscription

15. 7. EVPN VXLAN
✓ Configure loopbaks for VXLAN and EVPN.
✓ Configure L3 links between nodes.
✓ Configure Equal-Cost Multi-Path routing (ECMP) with 3 links for the
3x10 Gbps connection.
✓ Configure BGP.
✓ Check peerings.
✓ Check routes.
✓ Check flood lists (VXLAN).
✓ Do failover tests with each link.
Equinix bitNAP
Campus Nord
Arista
7280SR
Arista
7280SR
Arista
7504R

16. 8. Options for the Upgrade between Nodes: Lab
1. Main link 100 Gbps + Backup 40 Gbps with ECMP.
1. Main link 100 Gbps, backup 40 Gbps.
2. Lower priority to the 40 Gbps using MED in BGP.
3. ECMP in the 40 Gbps link to balance the traffic and BFD to detect link failures.
2. One link with 140 Gbps with UCMP.
1. Main link 140 Gbps using 5 differents channels.
2. Weighted: it would be like logically “splitting” the 100 Gbps into 10x10 Gbps, so that
we would have 14 “slots” and each packet would take one of the 14 slots.
3. Hashing lets us fix that any pair of addresses use the same link in both directions
but it must be the same for all the links.
40 Gbps
100 Gbps
Arista
7280SR
Arista
7280SR
Arista
7504R
bitNAP
Campus Nord Equinix
100 Gbps
100 Gbps
Arista
7280SR
Arista
7280SR
Arista
7504R
bitNAP
Campus Nord Equinix
140 Gbps

17. 9. Upgrade between nodes
✓ Different latencies in the channels and the hashing change from L4 to L3
could be a potential issue for your LACP links (less balance).
✓ Main link 100 Gbps + Backup 40 Gbps with ECMP.
✓ We went for option 1:
40 Gbps
100 Gbps
Arista
7280SR
Arista
7280SR
Arista
7504R
bitNAP
Campus Nord Equinix
100 Gbps

18. 9. Upgrade between Nodes
✓ Main link 100 Gbps + Backup
4x10 Gbps with ECMP.
✓ Lower priority to the 40 Gbps
using MED in BGP.
✓ ECMP in the 40 Gbps link to
balance the traffic; BFD to
detect link failures.
✓ Upgraded and both links
tested without service
interruptions.
• 00:00: Services in maintenance mode.
• 00:01: Iperf test start
• 00:35: Iperf test ends without errors
• 00:46: 100Gbps OK
• 01:03: Failover OK. Switch to 4x10G
• 01:08: Failback OK. Back to 100 Gbps.
• 01:25: Tests OK.
• 01:30: Services out of maintenance.
bitNAP
Campus Nord
40 Gbps
100 Gbps
Arista
7280SR
Arista
7280SR
Arista
7504R
Equinix
100 Gbps

19. And Done!

20. “Horror Story” – A Migration with a Happy Destination
✓ ...or how you should never unplug the equipment, but the equipment
saved our day.
• Just before the migration of the peering platform, we moved equipment in
the lab to the final locations.
• Everything was already configured in the switches.
– Alice: “Can I unplug it?”
– Bob: “Sure, no problem”
– Switch: Factory default
✓ But the switches have a cool functionality: “CLI scheduler”: it
automatically saves the last 100 outputs of show tech-support (taken
every hour).

21. Thanks for your attention!
Questions?
mariaisabel.gandia@csuc.cat