Networking Many VPCs: Transit and Shared Architectures - NET404 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Connecting Many VPCs
S h a r e d a n d T r a n s i t A r c h i t e c t u r e s
N i c k M a t t h e w s , P a r t n e r S o l u t i o n s A r c h i t e c t
V i c e n t e D e L u c a , S t a f f S o f t w a r e E n g i n e e r , Z e n d e s k
D e c e m b e r 1 , 2 0 1 7
N E T 4 0 4
AWS re:INVENT

VPN
WAN
AWS Direct
Connect
Transit VPCVPN
WAN
AWS Direct
Connect
Transit VPC
Shared Services
VPC Peering
Authentication,
Security, Logging
Shared Services
VPC Peering
Authentication,
Security, Logging
Many VPCs

“A virtual network that
closely resembles a
traditional network that
you'd operate in your own
data center”
What is a Virtual Private Cloud (VPC)?
Instance
Availability Zone
Instance
Availability Zone

Traditional Network
VPN VPN
WAN
Fiber
Applications Applications

VPN VPN
(VPC Peering)
WAN
Fiber
(AWS Direct Connect)
Applications Applications
AWS Network

Ease of creation Access models Diverse ownership
VPC Management Differences

• Less accounts and networks to setup
• Tighter control within the account or
VPC
• Identity and Access
Management (IAM)
• Strict security groups and
routing
• Identifying resources with tags
• Billing and ownership complexity
• Larger account or VPC blast radius
• User privileges, AWS limits
• More accounts and infrastructure to setup
• Tighter control of provisioning and
standards
• Automation of infrastructure
• AWS Direct Connect and VPN
standards
• Subnet and routing standards
• Simpler billing
• Smaller blast radius for users and networks
• Larger blast radius for shared
infrastructure and services
S m a l l e r V P C s o r a c c o u n t sL a r g e r V P C s o r a c c o u n t s
Account and VPC Segmentation

• Automation of infrastructure
• AWS Direct Connect and VPN
standards
• Subnet and routing standards
• Identity and Access
Management (IAM)
• Strict security groups and
routing
• Identifying resources with tags
S m a l l e r V P C s o r a c c o u n t sL a r g e r V P C s o r a c c o u n t s
Account and VPC Segmentation
Infrastructure and NetworkingPolicy and IAM

Related Sessions
Breakout Sessions:
ENT323: Enabling Governance, Compliance, and Operational and Risk Auditing with AWS Management Tools
ENT324: Automating and Auditing Cloud Governance and Compliance in Multi-Account Environments
SID206: Best Practices for Managing Security Operation on AWS
SID314: IAM Policy Ninja
SID331: Architecting Security and Governance Across a Multi-Account Strategy
Chalk Talks:
SID208: Less (Privilege) Is More: Getting Least-Privilege Right in AWS
SID308: Multi-Account Strategies
SID309: Credentials, Credentials, Credentials, Oh My!
Workshops:
NET308: VPC Design Scenarios for Real-Life Use Cases
SID311: Designing Security and Governance Across a Multi-Account Strategy

What to Expect
Compare VPC designs that:
• Scale
• Connect multiple VPCs together
• Provide automation
With these design patterns:
Transit VPC
VPN
WAN
AWS Direct
Connect
Transit VPC
Transit VPC
with firewalls
AWS Direct
Connect
WAN
Shared Services Multi-Region
Options
VGW
VGW
VGW
VGW
VGW
VGW
VGW
VGW

Our Starting Point
VPN
WAN
AWS Direct
Connect
Virtual private
gateway
Dev Prod

Challenge: Adding more VPCs
VPN
WAN
AWS Direct
Connect
Lots of connections
Dev Prod Dev Prod Dev Prod

Challenge: Peering VPCs
VPN
WAN
AWS Direct
Connect
VPC to VPC connections?

Challenge: Peering VPCs
VPN
WAN
AWS Direct
Connect
Connect Dev and Prod
VPC Peering
Connect the blue environment
How does this scale?
Let’s:

VPN
WAN
AWS Direct
Connect
Scaling connections?
Scaling VPC peering?

Transit VPC VPN
WAN
AWS Direct
Connect
Transit VPC

Familiarity and visibility Firewall insertion Encryption everywhere
Centralization Higher scale Inter-region connectivity
Benefits of the Transit VPC
VPN
WAN
AWS Direct
Connect
Transit VPC

VPN
WAN
AWS Direct
Connect
Transit VPC
Transit VPC
Architecture

Transit VPC: Hub
Availability Zone 1
Subnet 1
VPN Instance
Availability Zone 2
Subnet 2
VPN Instance
• Instances running VPN software
• Deployed in two Availability Zones
Internet gateway

Transit VPC: Routing
Virtual Private
Gateway (VGW)
VGW
VGW
Virtual Private
Network (VPN)
Border Gateway
Protocol (BGP)
Transit VPC
10.0.0.0/16
10.1.0.0/16
The VGW advertises the VPC CIDR to the
VPN instance (10.1.0.0/16)
Customer Gateway
(CGW)
So far, this works exactly like a typical VPN

Transit VPC: Routing
Virtual Private
Gateway (VGW)
VGW
VGW
Virtual Private
Network (VPN)
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 VGW
Internet
The VPN Instances
advertise routes to each
VGW. This can be a default
route or individual routes.

Why Doesn’t Peering Work?
VPC Peering
Transit VPC
10.0.0.0/16
10.1.0.0/16 10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 PCX
Internet

Why Doesn’t Peering Work?
VPC Peering
Transit VPC
10.0.0.0/16
10.1.0.0/16 10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 PCX
Internet
Destination: Internet Traffic must either originate or
terminate on a network
interface in the VPC
Transitive Routing

Why Does VPN Work?
Transit VPC
10.0.0.0/16
10.1.0.0/16 10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 VGW
Internet
Destination: Internet
VGW
VGW
Virtual Private
Network (VPN)
VGW
VGW
Traffic must either originate or
terminate on a network
interface in the VPC
Transitive Routing

Transit VPC: Availability
Virtual Private
Gateway (VGW)
VGW
VGW
Virtual Private
Network (VPN)
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 VGW
BGP and Dead Peer Detection (DPD)
detect the failure
The VGW route automatically fails
over to the other tunnel
Internet

Transit VPC: Availability
VGW
VGW
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/0 VGW
Route Table
Destination Target
10.2.0.0/16 Local
0.0.0.0/0 VGW
detect the failure
detect the failure
Internet

Options for On-Premises Connectivity
Transit VPC
Internet
On-premises
Virtual Private
Network (VPN)
Internet Gateway
WAN
Virtual Private
Network (VPN)
VGW
VGW
Detached
VGW
AWS
Direct
Connect
Customer
Gateway
VPN over the internet
WANVGW
VGW
Customer
Gateway
VGW
Virtual Private
Network (VPN)
AWS
Direct
Connect
VPN over AWS Direct
Connect
Detached VGW with AWS
Direct Connect

VPN Over the Internet
VGW
VGW
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Internet
172.16.0.0/16
On-premises
Virtual Private
Gateway (VGW)
Virtual Private
Network (VPN)
Virtual Private
Network (VPN)
• Use this design for more
control and visibility
• Supports alternative tunnels
such as DMVPN and GRE
• Manually configured and
operated
+
-

VPN Over AWS Direct Connect
VGW
VGW
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Virtual Private
Gateway (VGW)
Virtual Private
Network (VPN)
WAN
172.16.0.0/16
On-premises
Virtual Private Network (VPN)
VGW
VGW
Customer
Gateway
(CGW)
VGW
• Useful for encrypting
connections or inserting services
• More control over latency and
quality of the connectivity
• Supports alternative tunnels
such as DMVPN and GRE
• Manually configured and
operated
AWS
Direct
Connect
Private Virtual
Interface (VIF)
+
-

Detached VGW with AWS Direct Connect
VGW
VGW
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Virtual Private
Gateway (VGW)
Virtual Private
Network (VPN)
WAN
172.16.0.0/16
On-premises
VGW
VGW
Customer
Gateway
(CGW)
Detached
VGW AWS
Direct
Connect
Private Virtual
Interface (VIF)

Detached VGW with AWS Direct Connect
VGW
VGW
Transit VPC
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
WAN
172.16.0.0/16
On-premises
Virtual Private
Gateway (VGW)
Customer
Gateway
(CGW)
Detached
VGW
On-premises looks like another spoke
AWS
Direct
Connect
VGW
VGW
• Use this design if consistency
and automation are important
• Less management overhead
• Traffic can take multiple
routes out from AWS
• Traffic is unencrypted on the
private network
+
-

Transit VPC
Automation
AWS CloudFormation and
Implementation Guide

Transit VPC: Hub
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
• Uses the Cisco CSR
• Available in BYOL or Hourly billing
from AWS Marketplace
• Full featured IOS-XE device
• Deployed in two Availability Zones
• Support for duplicate tunnel addresses
Internet gateway
https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws_transitVPC/b_csraws_transitVPC_chapter_01.html

AWS Region
Transit VPC
S3 Bucket
for
VPN
Config
Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
100.64.127.224 / 27
Transit
VPC:
Creation
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
EC2 Auto-recovery

Transit VPC
S3 Bucket
for
VPN Config
Spoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = true
Transit VPC:
Add Spoke
SSH Only to CSR Security Group
VGW
AWS Key
Management
Service (KMS)
AWS Region
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR

Transit VPC
S3 Bucket
for
VPN Config
Spoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = falseVGW
AWS Region
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
Transit VPC:
Remove Spoke

AWS Lambda
Cisco
Configurator
AWS Lambda
VGW
Poller
Transit VPC:
Add Spoke in
Another Region
AWS Region
S3 Bucket
for
VPN Config
Transit VPC
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
AWS Region

AWS Lambda
Cisco
Configurator
AWS Region
Transit VPC
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
AWS Region
AWS Account
1. Setup the VGW
Poller
2. Allow bucket access
3. Allow KMS access
Transit VPC:
Add Spoke in
Another Account S3 Bucket
for
VPN Config
AWS Key
Management
Service (KMS)
AWS Lambda
VGW
Poller

AWS Account
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
AWS Region
Transit VPC
AWS Region
S3 Bucket
for
VPN Config
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW
Poller
Transit VPC:
Add Spoke in
Another Account
AWS Key
Management
Service (KMS)
Launch CloudFormation in Spoke Account
1. Setup the VGW
Poller
3. Allow KMS access

S3 Bucket
for
VPN Config
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW
Poller
Transit VPC:
Add Spoke in
Another Account
AWS Key
Management
Service (KMS)
1. Setup the VGW
Poller
3. Allow KMS access
AWS Region
Availability Zone 1
Subnet 1
Cisco CSR
Availability Zone 2
Subnet 2
Cisco CSR
Transit VPC
AWS Region
AWS Account

{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<account-1-ID>:root",
"arn:aws:iam::<account-2-ID>:root"
]
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::<S3 bucket name>/<bucket prefix>/*"
}
]
}
S3 Bucket Policy for a Spoke Account
One additional account can be defined at launch

{
"Sid": "Allow use of the key",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam:: <transit-vpc-primary-account-id>:role/TransitVPC-TransitVpcPollerRole-[cloudformation-id]",
"arn:aws:iam:: <transit-vpc-primary-account-id>:role/TransitVPC-CiscoConfigFunctionRole-[cloudformation-id]",
"arn:aws:iam:: <transit-vpc-primary-account-id>:role/TransitVPC-LambdaLoaderRole-[cloudformation-id]",
"arn:aws:iam::<account-1-id>:root",
"arn:aws:iam::<account-2-id>:root"
]
},
Key Management System: Key Policy
One additional account can be defined at launch

Firewalls and the
Transit VPC
VGW
VGW VGW
VGW
Internet

Firewall Use Cases
“We need a firewall for all traffic between on-premises and AWS.”
“We have compliance requirements for intrusion detection in our VPCs.”
“Our security organization requires application-level inspection.”
“We would like to centralize security appliances for any internet traffic.”

Transit VPC: Security Services
VGW
VGW
Virtual Private
Network (VPN)
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
Active/Passive
AS-path prepend

VPN Configuration
Transit VPC: Duplicate Tunnel Addresses
VGW
VGW
10.0.0.0/16
10.1.0.0/16
VGW
VGW
10.2.0.0/16
edit network interface tunnel units tunnel.2
set ip 169.254.45.210/30
set mtu 1427
top top
edit network virtual-router default protocol bgp
set router-id 52.1.1.1
set install-route yes
set enable yes
set local-as 65000
edit peer-group AmazonBGP
edit peer amazon-tunnel-vpn-30c1d051-1
set peer-as 7224
set connection-options keep-alive-interval 10
set connection-options hold-time 30
set enable yes
set local-address ip 169.254.45.210/30
set local-address interface tunnel.2
set peer-address ip 169.254.45.209
top
edit network interface tunnel units tunnel.2
set ip 169.254.45.210/30
set mtu 1427
top top
edit network virtual-router default protocol bgp
set router-id 52.1.1.1
set install-route yes
set enable yes
set local-as 65000
edit peer-group AmazonBGP
edit peer amazon-tunnel-vpn-30c1d051-1
set peer-as 7224
set connection-options keep-alive-interval 10
set connection-options hold-time 30
set enable yes
set local-address ip 169.254.45.210/30
set local-address interface tunnel.2
set peer-address ip 169.254.45.209
top
Same Address

Define Your Own Tunnel Address
Math time: Use after creating 10-15 VPNs to the same device

Instance-Based Transit VPC
Virtual Private
Network (VPN)
Spoke VPC Route
Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 ENI
• Use this design for more control
of both sides of the connection
• A wider set of solutions can be
used, like firewalls
• Failover and management at
scale can be challenging
• Use more than one device in
each VPC for better availability
• Instances inside spoke VPCs can
be intrusive
• No dynamic routing with BGP to
the route tables
+
-

Spoke VPC Route
Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 ENI
• Design is highly scalable
• Design for failover in each
Availability Zone
• Requires centralized
management
• Devices must support many
tunnels
• Licensing costs may be
prohibitive
• High management overhead
• Route propagation is more
difficult
VPN
Mesh
Full-mesh VPN
Connectivity
+
-
Bandwidth
Licensing

Transit VPC ecosystem
AWS VPN in Spoke VPCs
Instance VPN
in Spoke VPCs
Continuous Automation
One-time Automation
Manual Deployment
Routing
Firewalls
Anything with
BGP and VPN
Anything with
BGP and VPN
vSRX
CSR
Anything with
tunnels
vMX

Transit VPC:
Cost, Scale, and
Performance

Transit VPC Costs
VPN
WAN
AWS Direct
Connect
Transit VPC
Per Spoke:
VPN Hourly Charges
VPN Egress Charges
Hub Traffic:
Spoke Destination Egress
Egress Charges
Transit Instances:
Amazon EC2 Charges
Licensing Costs

Reference: Cisco CSR Bandwidth Tests
Size Routing
(Mbps)
1500B
VPN (Mbps)
1400B
T2.medium 390 300
M3.medium 300 250
C4.large 575 550
C4.xlarge 860 860
C3.2xlarge 1330 1000
C4.2xlarge 2300 2200
C4.4xlarge 4600 4100
C4.8xlarge 5000 4700
Note: Large packets

VPN
WAN
AWS Direct
Connect
Transit VPC
Transit VPC Performance
Use Security Groups inside the VPC
Use VPC Peering better performance
Each VPN instance can forward ~1-
3 Gbps aggregate on traffic and
instance size
Then what?
Each spoke can forward ~1.25 Gbps
per VPN tunnel
At 100 spokes, aggregate routes

Scaling the Transit VPC
VPN
WAN
AWS Direct
Connect
Transit VPC

Scaling the Transit VPC
VPN
WAN
AWS Direct
Connect
Transit VPC
VPN
WAN
AWS Direct
Connect
Transit VPC
Use pods of independent Transit VPCs
Connect the pods with tunnels for East-West traffic
VPN

AWS Direct Connect
for Many VPCs
WAN

AWS Direct Connect to Many VPCs
AWS Region
VGW
VGW
10.1.0.0/16
WAN
On-premises
AWS Direct Connect
Location
Private Virtual Interface
(VIF)
Customer
Router
AWS
Router
Customer
Router
AWS
Router
VGW
VGW
10.2.0.0/16
Up to 50 VIFs per port
AWS Direct Connect
Location 2

AWS Direct Connect: Link Aggregation
AWS Region
VGW
VGW
10.1.0.0/16
WAN
On-premises
Link Aggregation
(LAG)
Private Virtual Interface
(VIF)
Customer
Router
AWS
Router
Customer
Router
AWS
Router
VGW
VGW
10.2.0.0/16
Up to 4 ports in a LAG,
each with 50 VIFs
AWS Direct Connect
Location 2

Direct Connect Gateway
AWS Region
VGW
VGW
10.1.0.0/16
WAN
On-premises
AWS Direct Connect
location
Private Virtual
Interface (VIF)
Customer
Router
AWS
Router
Customer
Router
AWS
Router
VGW
VGW
10.2.0.0/16
Up to 10 VGWs per direct
connect gateway
AWS Direct Connect
location 2
Direct
connect
gateway
Account

Multiple Regions
WAN
On-premises
AWS Direct Connect
Location
Private Virtual
Interface (VIF)
Customer
Router
AWS
Router
Customer
Router
AWS
Router
AWS Region
VGW
VGW
VGW
VGW
AWS Direct Connect
Location 2
Direct
connect
gateway
Account
AWS Region
VGW
VGW
VGW
VGW

Shared Services VPCs

Shared Services VPC
• Authentication
• Logging
• DevOps tools
• Security resources
• Deployed in each AWS Region
Shared Services
VPC Peering
Authentication,
Security, Logging

VPC Peering
Challenges
VPN
WAN
AWS Direct
Connect
Shared Services
VPC Peering
Full VPC connectivity
172.16.0.0/16 172.16.0.0/16
No overlapping addresses
…125
Scale

Introducing: PrivateLink
Shared
Services VPC
10.1.0.0/16
10.1.1.0/24
Availability Zone
10.1.1.127
10.1.2.0/24
Availability Zone
10.1.2.35
172.16.0.0/16
172.16.1.0/24
Availability Zone
172.16.2.0/24
Availability Zone
Network Load
Balancer
API API
One IP Address for each
Availability Zone
The endpoint is a local IP address
Access is unidirectional
172.16.1.9 172.16.2.41

Introducing: PrivateLink
Shared
Services VPC
10.1.0.0/16
10.1.1.0/24
Availability Zone
10.1.1.127
10.1.2.0/24
Availability Zone
10.1.2.35
172.16.0.0/16
172.16.1.0/24
Availability Zone
172.16.2.0/24
Availability Zone
172.16.1.9 172.16.2.41
API API
10.1.0.0/16
10.1.1.0/24
Availability Zone
10.1.1.162
10.1.2.0/24
Availability Zone
10.1.2.22
Support for
overlapping IP address
ranges
…thousands

VPN
WAN
AWS Direct
Connect
Transit VPC
Shared
Services
Transit VPC with Services VPC:
It’s just a spoke

Shared Services Options
VPN
WAN
AWS Direct
Connect
Shared Services
VPC Peering
VPC Peering
• Bidirectional services
• Broad VPC access
• No load balancers required
+
PrivateLink
• Unidirectional services
• More granular access
• Scale beyond 125 spokes
• Overlapping addresses
+
Transit VPC
• Transit VPC consistency
• Automation is built-in
• Lower performance
• More complex to manage
+
VPN
WAN
AWS Direct
Connect
Transit VPC
Shared
Services
-

Multiple Regions
AWS Region
VGW
VGW
VGW
VGW
AWS Region
VGW
VGW
VGW
VGW

Inter-Region VPC Peering
AWS Region AWS Region
VPC
Peering

Global Connectivity Options
VPN
WAN
AWS Direct
Connect
Transit VPC
Cross-Region VPNDirect Connect Gateway Inter-Region Peering
• Native connectivity
• No management required
• One-to-one VPC
configuration
+• AWS Direct Connect only
• High performance
• No management required
+ • Transit VPC consistency
• Full control of connectivity
• Lower performance
• More complex to manage
+
--

VPC Inflection Points
VPN Direct Connect VPC
5
15
50
100
125
200+
Transit VPC
Define your own
tunnel addresses
Automation Automation
VIF to port Limit Route and peering limit
Max route limit
Max peer limit
Limit on 4x LAG
Transit VPC
Customization
1

200+
Internet
Secure protocols
Secure authentication
Bastion hosts
Use PrivateLink

Tips and Tricks

Advice
• Networking changes fast, no more crystal balls
• Segment as needed
• Experiment and test
• Mix and match! These are starting points not dogma!

Upgrading or modifying the Transit VPC
• Resize:
1. Stop the instance
2. Choose a different size
3. Start the instance
• Cisco CSR 16.5.1b (Apr ’17) and later support inline .BIN upgrades
• Same upgrade procedure as typical Cisco upgrades
• Before 16.5.1b you must use new Amazon EC2 instances
• Reuse existing Elastic IP Addresses and CLI configuration
• Hourly to BYOL also requires new instances

Customizing the Transit VPC
• Two AWS Lambda functions:
• https://<S3region>/solutions-reference/transit-vpc/v4/transit-vpc-poller.py1
• https://<S3region>/solutions-reference/transit-vpc/v4/transit-vpc-push-cisco-
config.zip1
• Use cases
• Create your own tags and policies
• Customize BGP policy
• Advertise different routes
• Aggregate or 0.0.0.0/0 after 100 routes
• Cisco DMVPN and other features
1: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

Interconnecting
VICENTE DE LUCA, STAFF SOFTWARE ENGINEER, ZENDESK
vdeluca@zendesk.com

About us
Zendesk builds software for better customer relationships

Traditional backbone architecture
• Multiple providers
• BGP
United
States
Europe
2015 Zendesk internal backbone
MPLS
High-availability
Multiple circuits
United
States
Europe

Transitioning to a Hybrid Model
VPC Managed VPN
over Internet and
public peering
2 Firewalls per data center
active/standby
VPN
High Availability Best Practices
2 IPsec tunnels per VPN connection
active/standby
16 IPsec Tunnels: 4 active
16 BGP peers:
- AS prepend
- Local preference
VGW
VGW

VPC: One more, please !
EC2 VPN instances
Complex environment:
Configuration changes touch
all environments
33 IPsec Tunnels: 9 active
VGW
VGWVGW
VGW
VPN

Fast growing
VPN
VGW
VGW
VPN
VGW
VGW
VPN
VGW
VGW
Early 2017 Zendesk internal backbone
VGW
VGW

Proposed solution and requirements
Build a location agnostic network. Make it simple, cost-effective, and quick to deploy.
Project Medusa
Provides full-mesh encrypted dual stack connectivity
Support on-premises data centers and multi-cloud providers
Scalable for growth: bandwidth / packets per second
Describe the network stack as code
Fully automated – bootstrap, scaling and self-healing
Kubernetes Pods native routing – no NAT !

S h a r e d S e r v i c e s V P C s a n d D y n a m i c M u l t i - p o i n t V P N
Shared Services VPC:
• Authentication, Service Discovery, Logging
• CSR1000V as DMVPN Hubs
What’s in the DMVPN standard?
• IPsec
• Generic Routing Encapsulation (GRE)
• Next Hop Resolution Protocol (NHRP)
• Internal BGP (iBGP)
LDAP
Master
Consul
Vault
Splunk LDAP
Master
Consul
Vault
Splunk
US-EAST
EU-CENTRAL
DMVPN
HUBs

A r c h i t e c t u r e : C o n n e c t a d a t a c e n t e r
DATA CENTER
DMVPN SPOKE
• Deploy 2x Routers (IOS-XE)
• Standard DMVPN configuration
• Spokes: Hubs are statically configured
• Hubs: No change required for new spokes
• Initially a Hub and Spoke topology
• iBGP with additional paths extension
• Equal Cost Multipath (ECMP) active/active
LDAP
Master
Consul
Vault
Splunk LDAP
Master
Consul
Vault
Splunk
US-EAST
EU-CENTRAL
DMVPN
Spoke
Data center 1
DMVPN
HUBs

A r c h i t e c t u r e : C o n n e c t a V P C
LDAP
Master
Consul
Vault
Splunk LDAP
Master
Consul
Vault
Splunk
US-EAST
EU-CENTRAL
DMVPN
HUBs
AZ 1 (public) AZ n (public)
ASG
Medusa Routers
ASG
Medusa Routers
DMVPN
Spokes
?
VPC ACME
DMVPN SPOKE
Medusa Routers:
• EC2 Instance: C4.xlarge + Ubuntu 17
• Auto scaling group (ASG): min. 2 instances per AZ
• DMVPN: Racoon (IPsec) + OpenNHRP + BIRD (BGP)
• Kernel: >=4.12 for ECMP Layer4 hash
• Custom: Python(boto3): AWS Routing Table BGP
propagation
Motivations:
• Linux portability
• Cost-effective (open source)
• Tooling ecosystem integration
• Flexible foundation

A r c h i t e c t u r e : H o w d y n a m i c V P N t u n n e l s a r e b u i l t
1. First packet from Spoke A to Spoke B is forwarded to a hub router
The hub forwards the packet to Spoke B (+latency)
2. The hub sends a redirect message to Spoke A with Spoke B’s public address
3. Spoke A initiates a dynamic IPsec tunnel to Spoke B
Further communication takes the new direct IPsec tunnel Aó B
NHRP
Hub
NHRP
Spoke A
NHRP
Spoke B
IPsec

S c a l i n g N x V P C s u s i n g D M V P N
LDAP
Master
Consul
Vault
Splunk LDAP
Master
Consul
Vault
Splunk
US-EAST
EU-CENTRAL
AZ 1 (public)
ASG
Medusa Routers
ASG
Medusa Routers
AZ 1 (public)
ASG
Medusa Routers
ASG
Medusa Routers
AZ n (public)AZ n (public)

A r c h i t e c t u r e : B u i l d i n g t h e C l o u d S t a c k s
VPC and infrastructure:
• Provisioned using Hashicorp Terraform
Medusa Router:
• Custom AMI built with Packer
• Linux bootstrap: bash + python scripts
• Terraform module
New VPC required configuration:
1. Name
2. Environment
3. AWS Region
4. Availability Zones
5. VPC CIDR
6. VPC Public/Private/Custom subnets
terraform apply && echo NetOps rocks!

D e p l o y m e n t s t a t u s , D I Y c h a l l e n g e s a n d t h e f u t u r e
• Gradually cutting over production from legacy VPN to Medusa
• 4 data centers, 12 VPCs into 6 regions and 2 cloud providers
Performance tests:
• us-east ó us-west (latency avg. 73ms)
• TCP uni-directional throughput: ~3.5Gbps
(iperf: 20 connections, 4 Medusa Routers each VPC)
• Legacy VPN: ~400Mbps
DIY challenges:
• Router bootstrap requires a next available IP function: use IPAM or Lambda
• VPC Routing Table limit: propagate rfc1918 summary routes + exceptions
• Kernel >=4.12 bug in IPsec hw offload: kernel bugzilla id:197513 w/ patch
2018 Q1:
• Stay tuned at Zendesk Engineering blog. NetOps will publish an article in detail
• https://medium.com/zendesk-engineering

Take This Home
• Transit VPC for centralizing VPN configuration
• AWS Direct Connect can be used in addition to VPN
• Automation is important
• There are many options, and they change over time
• Customize, mix and match

Available February 2018
Networking Study Guide

Thank you!
Pl e ase su b m i t yo ur f e e db ac k!

Networking Many VPCs: Transit and Shared Architectures - NET404 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Networking Many VPCs: Transit and Shared Architectures - NET404 - re:Invent 2017

Similar to Networking Many VPCs: Transit and Shared Architectures - NET404 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Networking Many VPCs: Transit and Shared Architectures - NET404 - re:Invent 2017