As more customers adopt Amazon VPC architectures, the features and flexibility of the service are squaring off against evolving design requirements. This session follows this evolution of a single regional VPC into a multi-VPC, multi-region design with diverse connectivity into on-premises systems and infrastructure. Along the way, we investigate creative customer solutions for scaling and securing outbound VPC traffic, securing private access to Amazon S3, managing multi-tenant VPCs, integrating existing customer networks through AWS Direct Connect, and building a full VPC mesh network across global regions.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rob Alexander, Principal Solutions Architect
December 2, 2016
ARC302
From One to Many
Evolving VPC Design

2. Disclaimer:
Do Try This at Home!

3. Assuming you’ve heard of…
Route Table
Elastic
Network
Interface
Amazon VPC
Internet
Gateway
Customer
Gateway Virtual
Private
Gateway
VPN
Connection
VPC subnet
Network ACL
Security group
Enhanced
Networking
VPC
Peering
AWS Direct
Connect

4. Related Sessions
NET201 – Creating Your Virtual Data Center: VPC
Fundamentals and Connectivity Options
NET305 – Extending Datacenters to the Cloud:
Connectivity Options and Considerations for Hybrid
Environments

5. From one…
Subnet
Availability Zone A
Subnet
Availability Zone B
VPC

6. us-east-2
VPC
VPC
VPC
VPC
Transit VPC
VPC
us-west-2
VPC
VPC
VPC
eu-west-1
VPC
VPC
VPC
VPC
Transit VPC
VPC
Branch Branch
NA
HQ
VPC
VPC
VPC
VPC
VPC
VPC
Chicago DX
AP
HQ
London DX
ap-northeast-1
VPC
VPC
VPC
VPC
Transit VPC
VPC
EU
HQ
Tokyo DX … to many

7. VPC
/16
Choose a CIDR
• CIDR fixed on VPC
creation
• /16 down to /28
• Go Big

8. VPC IPv4 space design
• Plan for expansion to additional Availability
Zones or regions
• Consider connectivity to corporate networks
• Don’t overlap IP space
• Save space for the future
• IPv4 space is required, but …

9. IPv6 now supported in VPC
• Optionally enable IPv6 on VPC
• /56 of Amazon’s Global Unicast Address (GUA) per VPC
• /64 CIDR block per subnet
• IPv6 completely independent from IPv4
• Enabled per subnet or per instance (per ENI)
• Supported by Security Groups, Route Tables, NACLs, VPC
Peering, IGW, DX, Flow Logs and DNS Resolution

10. Availability Zone A
VPC
• Even distribution of IP
space across AZs
• Use at least 2 AZs
• Subnets are AZ
specific
• How big? How many?
Create subnets
Subnet
Availability Zone B
Subnet
Availability Zone C
Subnet
/16

11. Availability Zone A
Subnet
VPC
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet Subnet Subnet Subnet Subnet Subnet
Subnet Subnet Subnet Subnet Subnet
Subnet Subnet Subnet Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
Subnet
/16

12. VPC subnet design
• Traditional switching limitations do not apply
• Consider large, mixed use subnets
• Use security groups to enforce isolation
• Use tags for grouping resources
• Use subnets as containers for routing policy

13. Related Sessions
NET401 – Another Day, Another Billion Packets

14. Availability Zone A
Public subnet
Private subnet
Availability Zone B
Public subnet
Private subnet
VPC
/16
Availability Zone C
Public subnet
Private subnet
/22 /22 /22
/20 /20 /204091 IPs
1019 IPs

15. VPC
/16
Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
Availability Zone C
Private subnet
Public subnet
Private subnet
/22 /22 /22
/20
/20
/20
/20
/20
/20

16. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
.1
VPC
.1
.1 .1
.1 .1
Routing Policy
Main Route Table
Destination Target
10.1.0.0/16 Local
VPC CIDR 10.1.0.0/16

17. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 IGW

18. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW

19. Availability Zone A
Public subnet
Private subnet
Availability Zone B
VPC
/54
Availability Zone C
/64
/64
18 MILLION,
Public subnet
Private subnet
Public subnet
Private subnet
What about IPv6?
/64
/64
/64
/64
TRILLION
IPs

20. Availability Zone A
Private subnet
Public subnet
Availability Zone B
Private subnet
Public subnet
VPC
IPv6
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
2001:db8:1234:1a00::/56 Local
0.0.0.0/0 IGW
::/0 IGW

21. Availability Zone A
Private subnet
Public subnet
Availability Zone B
Private subnet
Public subnet
VPC
IPv6
Routing Policy
AWS Region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
2001:db8:1234:1a00::/56 Local
Corp CIDR VGW
::/0 EIGW
Egress-Only IGW

22. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 ???
Corp CIDR VGW

23. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Why go outside?
• AWS API endpoints
• Regional services
• Third-party services

24. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Instance
Corp CIDR VGW
NAT
Instance

25. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
Routing Policy
AWS Region
Internet
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Instance
Corp CIDR VGW
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 Black Hole
Corp CIDR VGW
NAT
Instance

26. Scalable and Available NAT

27. Evolving design requirements
• Public subnets for resources reachable from Internet
• Private subnets with egress only access to public network
• Scalable, highly available NAT
• One AWS account
• One VPC
• One region

28. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
NAT
Instance
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 Black Hole
Corp CIDR VGW

29. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Gateway
Corp CIDR VGW
NAT
Gateway

30. Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
NAT Instance
Source IP: Port NAT’d Source IP:Port
Security Updates
Package Repos
NTP
VPC
Public Network

31. Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
Source IP: Port NAT’d Source IP:Port
VPC
Source IP is the same
Source Port must be
unique
Destination
IP and Port
are the same
NAT Instance
Public Network
52.27.192.88:33622
52.27.192.88:38438
52.27.192.88:48132
52.27.192.88:29754
Security Update

32. Why a NAT Gateway?
10.1.1.112:54318 52.27.192.88:35678
Source IP: Port NAT’d Source IP:Port
VPC
Source IP is the same
Source Port must be
unique
Destination
IP and Port
are the same
Public Network
52.27.192.88:33622
52.27.192.88:38438
52.27.192.88:48132
52.27.192.88:29754
NAT Gateway
Security Update

33. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
• Still need IGW
• Separate subnets
• Requires EIP
• AZ specific
• Burst to 10 Gbps

34. 1
NAT Gateway: Securing Access
NAT Gateway ENI:
Network ACL
Public subnet
NAT
Gateway
Network ACLs still apply

35. NAT Gateway: Securing Access
Use routing
policy to control
access to NAT
Gateway
Private subnet
Public subnet
Private subnet
NAT Enabled
no-NAT
no-NAT Private Route Table
Destination Target
10.1.0.0/16 Local
NAT Enabled Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT Gateway
NAT
Gateway
2

36. NAT Gateway: Securing Access
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 0.0.0.0/0
Use security groups
to restrict outbound
access for instances
Default VPC security group:
3

37. NAT Gateway: Securing Access
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 10.2.0.0/16
Outbound Rules
Type Protocol Port Range Destination
All traffic All 0 - 65535 0.0.0.0/0
Use security groups
to restrict outbound
access for instances
Default VPC security group:
NAT Enabled VPC security group:
3

38. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
NAT Enabled
no-NAT
NAT Enabled
no-NAT

39. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC
AWS Region
Internet
Deploy a
NAT Gateway
NAT
Gateway
NAT
Gateway
NAT Enabled
no-NAT
NAT Enabled
no-NAT

40. • Drop in replacement for NAT instance
• Fully managed
• Highly available and fault tolerant
• Scalable to 10 Gbps burst per gateway
• Supports VPC Flow Logs
• No higher level functions like IPS, UTM,
URL Filtering, packet inspection, etc
• Cannot associate security group to
gateway
Pro & Con: NAT Gateway

41. AWS
Region
Considering multiple VPCs
Public-facing
web apps
Internal
company
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
network

42. One VPC, Two VPC

43. VPC
Why not 1 big VPC?
Why not 1 AWS Account?
• Blast radius
• Account Limits
• API Limits

44. Considerations for one or many VPCs
AWS Region
Prod
Not
Prod
VPCVPC

45. Considerations for one or many VPCs
AWS Region
PCI
Apps
VPC VPC
Non
Regulated
Apps

46. Considerations for one or many VPCs
AWS Region
Prod
VPC
AWS Region
Disaster
Recovery
VPC

47. Considerations for one or many VPCs
AWS Region
VPC
Audit
Logging &
Analytics
AWS
CloudTrail
AWS
Config
VPC Flow
Logs
VPC
Legal
VPC
Finance
VPC
Sales
App Logs,
S3 Access Logs,
ELB Logs
Amazon
Redshift
Amazon
EMR
S3

48. AWS Region
Internal application to VPC
Public-facing
web app
Internal
company
app
VPN
connection
VPCVPC
Customer
network

49. Availability Zone A
Private subnet Private subnet
AWS Region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
Internal customers
Private Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
VPC
Internal application to VPC
Customer
network

50. But apps will make heavy use of …
Amazon S3
…as a primary data store

51. VPC Egress Control

52. Evolving design requirements
• VPN connectivity to private-only VPC
• No egress in the VPC to public networks
• Private IP access to Amazon S3
• Content-specific access controls
• One AWS account
• One VPC
• One region

53. Availability Zone A
Private subnet Private subnet
AWS
Region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
You really don’t want to do this:
Amazon
S3
Internet
Customer
border router
Customer VPN
Internet
VPC
Customer
network

54. Availability Zone A
Private subnet Private subnet
AWS
Region
Virtual
Private
Gateway
Intranet
app
Intranet
app
Availability Zone B
So do this instead:
Amazon
S3
VPC
VPN
connection
VPC Endpoints
• No IGW
• No NAT
• No public IPs
• Free
• Robust access
control
Customer
network

55. Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Private subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
Prefix List for S3 us-west-2 VPCE

56. Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Public subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 IGW
Prefix List for S3 us-west-2 VPCE

57. Creating S3 VPC endpoint
Private subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 NAT Gateway
Prefix List for S3 us-west-2 VPCE
Public subnet
NAT
Gateway

58. Prefix lists
aws ec2 describe-prefix-lists
PREFIXLISTS pl-68a54001 com.amazonaws.us-west-2.s3
CIDRS 54.231.160.0/19
CIDRS 52.218.128.0/18
• Logical route destination target
• Dynamically translates to service IPs
• S3 IP ranges change over time
• S3 prefix lists abstract change

59. Prefix lists
… and use them in your outbound security group rules!

60. Private subnet
Controlling VPC access to Amazon S3
AWS Identity & Access
Management (IAM) policy
on VPCE:
VPC
{
"Statement": [
{
"Sid": "vpce-restrict-to-backup-bucket",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject”
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::backups-reinvent",
"arn:aws:s3:::backups-reinvent/*"]
}
]
}
Backups bucket?

61. Private subnet
Controlling VPC access to Amazon S3
S3 bucket policy:
VPC
From
vpce-bc42a4e5?
{
"Statement": [
{
"Sid": "bucket-restrict-to-specific-vpce",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::backups-reinvent",
"arn:aws:s3:::backups-reinvent/*"],
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-bc42a4e5”
}
}
}
]
}

62. Controlling VPC access to Amazon S3
Recap on security layers:
1. Route table association
2. VPCE policy
3. Bucket policy
4. Security groups with prefix list
Private subnet
VPC
1.
2.
3.
4.

63. Private subnet Private subnet
AWS
Region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps

64. Private subnet Private subnet
AWS
Region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps
Private subnet Private subnet
Private subnet
Logs Analytics

65. • Secure, highly scalable and highly
available access to S3
• Fine grained control of access to
content in S3 from VPC
• Control which VPCs/VPCEs can
access which S3 buckets
• No public IPs required, source IPs kept
private
• Bucket policy restricted to specific
VPCs (or VPCEs) will disable S3
Console access
• Requires Amazon DNS enabled on
VPC
Pro & Con: VPC Endpoints

66. AWS Region
Public-facing
web apps
Internal-
only
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
network
Customer Gateway
(CGW)

67. Shared Service Hubs

68. AWS
Region
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPCVPC
Customer
network
Public
apps
Internal
apps

69. AWS Region
VPC
HA VPN Pair
Availability Zone A
HA VPN
To
VPC
iBGP
eBGP
Customer CIDRs or Default Route
eBGP
AWS ASN 7224
Re-advertise VPC CIDR via IGP
VGW
VPC CIDR
Customer ASN (Public or Private)
CGW1 CGW2
VPN1
Tun1
VPN1
Tun2
Availability Zone A
VPN2
Tun1
VPN2
Tun2
Reuse your CGW Public IP
to connect to more VPCs
Customer
network
MED
MED

70. AWS
Region
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Customer
network

71. Evolving design requirements
• Centralize network connectivity to and from cloud
• Centralize management, security, and common services
• Account owners in control of own VPC resources
• Many AWS accounts
• Many VPCs
• One region

72. AWS
Region
VPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Hub and
Spoke
with
Peering
VPC
Shared
services
VPC
VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
VPC

73. Customer
network
AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
VPC peering
Shared services
10.2.22.0/24
10.1.11.0/24

74. AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
172.16.0.0/16 PCX-1
Edge-to-edge routing
Shared services
10.2.22.0/24
10.1.11.0/24
172.16.0.0/16
Customer
network

75. AWS Region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Proxy
subnets
10.2.0.0/1610.1.0.0/16
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/16 PCX-1
Edge-to-edge via proxy
PCX-1 10.2.22.0/24
Internal
ELB
Proxy
fleet
Internet
Public
services
S3
VPC
Customer
network
Proxy Route Table
Destination Target
10.1.0.0/16 local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
Proxy Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
0.0.0.0/0 IGW
S3 Prefix List VPCE

76. Customer
network
Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS Region
Internet
VPC
Auto Scaling
proxy
fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC

77. Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS Region
Internet
VPC
Auto Scaling
proxy
Fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Bastion
host
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC
Customer
network

78. Shared Services Hub: To-Do List
• Use IAM to restrict spoke AWS accounts from altering network
• Create a NetOps IAM role in all accounts:
https://aws.amazon.com/blogs/security/how-to-assign-permissions-using-new-aws-managed-policies-for-job-functions/
• Enable AWS CloudTrail, AWS Config, and VPC Flow Logs for all accounts
• Integrate CloudTrail with CloudWatch Logs and create alarms:
https://aws.amazon.com/blogs/aws/cloudtrail-integration-with-cloudwatch-now-available-in-four-more-regions

79. • Minimizes on premises network change
• Reduces latency, cost of cloud
applications accessing common services
• Provides spoke accounts control over
own resources
• But controls and secures egress traffic
from spokes
• Security Groups work across peers
• Cost and management of central proxy
layer
• Not a transparent proxy
• Configuring end devices to use proxy
• Restricted to HTTP/S
• No transitive networking
• Peering data transfer cost
Pro & Con: Shared Services Hub and Spoke

80. AWS Region
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hub
Prod hub
Data
services
hub

81. AWS Region
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hub
Prod hub
Data
services
hub
VPC
VPC

82. Customer
network
AWS Region
Availability Zone A
Private subnet
VPC
Availability Zone B
Private subnet
AWS Lambda
Amazon API Gateway
Elastic
Network
Interface
VPVPC
VPC
VPC
Prod hub
VPC
Internet
Hybrid
Serverless
Amazon
Aurora
Replica
Mobile Application VPC

83. Legacy
Apps
Customer
network
AWS Region
Availability Zone A
Private subnet
VPC
Availability Zone B
Private subnet
AWS Lambda
Amazon API Gateway
Elastic
Network
Interface
VPVPC
VPC
VPC
Prod hub
VPC
Internet
Hybrid
Serverless
Amazon
Aurora
Replica
Mobile Application VPC

84. us-east-2 region
VPC VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
eu-west-1 region
VPC VPC
VPC
VPC
VPC
VPC

85. VPC Mass Transit

86. Evolving design requirements
• Centralize and minimize network connections
• Allow end to end routing from cloud to existing networks
• Minimal operational overhead
• Leverage AWS network
• Many AWS accounts
• Many VPCs
• Many regions

87. Availability Zone A
Public subnet
VPC
Transit
VPC
Availability Zone B
Public subnet
AWS
Region
EC2 VPN EC2 VPN

88. Availability Zone A
Public subnet
VPC
Transit
VPC
Availability Zone B
Public subnet
EC2 VPN EC2 VPN
AWS
Region VPC
Spoke VPC
Transit VPC
VPC
Spoke VPC
VPC
Spoke VPC

89. AWS
Region
VPC
VPC
VPC
VPC
VPC
VPC
VPC
Transit VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
Branches
Transit
VPC

90. https://aws.amazon.com/answers/networking/transit-vpc/
Transit VPC

91. Transit VPC
Built using Cisco Cloud Services Router (CSR) 1000V
• Available on the AWS Marketplace
• A virtualized ASR with full IOS-XE software stack
• BYOL or Pay-as-you-Go license models

92. Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
100.64.127.224 / 27
Transit VPC:
Creation

93. What is EC2 Auto Recovery?
RECOVER Instance
Instance ID
Instance metadata
Private IP addresses
Elastic IP addresses
EBS volume attachments
Instance retains:
* Supported on C3, C4, M3, M4, P2, R3, T2, and X1 instance types with EBS-only storage
StatusCheckFailed_System
Amazon CloudWatch
per-instance metric alarm:
When alarm triggers?

94. Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
VPC
Spoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = true
Transit VPC:
Add Spoke
SSH Only to CSR Security Group

95. Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
AWS Region
Transit VPC
VPC
Spoke VPCTransit VPC:
Preferred
Route
Spoke VPC Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 VGW
Transit VPC Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
Active / Active

96. Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
AWS Region
Transit VPC
VPC
Spoke VPC
transitvpc:preferred-path = CSR1
Transit VPC:
Preferred
Route
Spoke VPC Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 VGW
Transit VPC Route Table
Destination Target
100.64.127.224/27 Local
0.0.0.0 IGW
Prefix List for S3 VPCE
Spoke VGW Tag
Active / Passive

97. Transit VPC: Preferred route spoke configuration
From CSR2:
!
address-family ipv4 vrf vpn-8a23d2e3
neighbor 169.254.35.57 remote-as 7224
neighbor 169.254.35.57 timers 10 30 30
neighbor 169.254.35.57 activate
neighbor 169.254.35.57 as-override
neighbor 169.254.35.57 soft-reconfiguration inbound
neighbor 169.254.35.57 route-map rm-vpn-8a23c7e3 out
exit-address-family
!
route-map rm-vpn-8a23c7e3 permit 10
set as-path prepend 64512 64512
!
BGP AS override
configured by default

98. Availability Zone A
Public subnet
VPC
Availability Zone B
Public subnet
CSR1 CSR2
AWS Region
Transit VPC
S3 Bucket
for
VPN Config
VPC
Spoke VPC
AWS Lambda
Cisco
Configurator
AWS Lambda
VGW Poller
transitvpc:spoke = false
Transit VPC:
Remove Spoke

99. AWS
Region
VPC
VPC
VPC
VPC
VPC
VPC
VPC
Transit VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
Branches
Transit
VPC
Internet
Public
services

100. Customer
network
VPC
Transit VPC
us-east-2
us-west-2
VPC
VPC
Spoke VPC
Spoke
VPC
VPC
Transit VPC
eu-west-1
eu-central-1
VPC
VPC
Spoke VPC
Spoke
VPC
AWS Network
Backbone
Internet

101. • End to End routing between VPCs in all
regions and any other non-AWS network
• Central transit routers can perform higher
level networking and security functions
• Spoke VGWs are HA by default
• Minimizes on premises networking changes
• Can minimize cost if replacing on premises
or colo networking hardware
• Availability and management of transit router
instances
• Licensing costs
• Cost of data transfer between transit, spokes
and other networks
Pro & Con: Transit VPC

102. AWS
Region VPC
VPC
VPC
VPC
VPC
Transit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Transit VPC
with
AWS Direct
Connect
(DX)
Detached
VGW
transitvpc:spoke = true
Customer
network
AWS Direct Connect
location
Private virtual interface (VIF) to
detached VGW
• 1 PVI per VGW
• 1 BGP ASN
• 1 802.1Q VLAN Tag
• 1 BGP MD5 key
Private fiber connection
One or multiple
50 – 500 Mbps,
1 Gbps or 10 Gbps pipes

103. AWS
Region VPC
VPC
VPC
VPC
VPC
Transit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Customer
network
AWS Direct Connect
location
Private DX VIF to
dedicated VGW
100.64.127.224 / 27
Private Virtual Interface 1
VLAN Tag 101
BGP ASN 7224
BGP Announce 100.64.127.224/27
Interface IP 169.254.251.5/30
Customer Interface 0/1.101
VLAN Tag 101
BGP ASN 65001
BGP Announce Customer Internal
Interface IP 169.254.251.6/30

104. AWS
Region VPC
VPC
VPC
VPC
VPC
Transit VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Customer
network
AWS Direct Connect
location
Public DX VIF to
dedicated VGW
Public EIPs
Public Virtual Interface 1
VLAN Tag 501
BGP ASN 7224
BGP Announce AWS Regional
Public CIDRs
Interface IP Public /30 Provided
Customer Interface 0/1.501
VLAN Tag 501
BGP ASN 65501 (or Public)
BGP Announce Customer Public
Interface IP Public /30 Provided
NAT + Security layer

105. Equinix Chicago
Customer
network
us-west-2
VPC
VPC
VPC
VPC
Transit VPC
VPC
us-east-2
VPC
VPC
VPC
VPC
Transit VPC
VPC
AWS Direct Connect Inter-Region Connectivity
A single DX Public interface can reach all US regions

106. • Be selective in your public network announcements
• Filtering public prefix announcements if necessary
• Authoritative AWS public IP list available:
https://ip-ranges.amazonaws.com/ip-ranges.json
• For notification of IP changes, subscribe to SNS topic:
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged
AWS Direct Connect Public Interface

107. Related Sessions
NET402 – Deep Dive: AWS Direct Connect and VPNs

108. Leverage corporate network
Headquarters
Branch
Branch
DX Location
Provider Edge (PE)Customer Edge (CE)
eBGP
Provider
MPLS
Network
PECE
PE
CE
eBGP
AWS Region
MPLS / IPVPN
PE DX
eBGP
CE PE

109. Headquarters
Branch
Branch
Chicago DX Location
eBGP
Provider
MPLS
Network
PECE
PE
CE
AWS
Ohio
region
Multi-region DX
PE DX
eBGP
CE PE
London DX Location
AWS
Ireland
region
PE DX
eBGP
Going global
AS 7224
AS 7224
100 BGP Route Max
100 BGP Route Max

110. • Private network, no Internet dependencies
• Predictable latency on DX connections
• Dedicated bandwidth to AWS
• Access to public networks of all US regions
over single US based DX connection
• Public DX BGP announcements may require
filtering
• For large networks, 100 route per VPC limit
may require summarization or default routes
• Cost of provider network and DX connections
Pro & Con: Transit VPC with DX

111. us-east-2
VPC
VPC
VPC
VPC
Transit VPC
VPC
us-west-2
VPC
VPC
VPC
eu-west-1
VPC
VPC
VPC
VPC
Transit VPC
VPC
AWS Network
Backbone
Provider
MPLS
Network
Branch Branch
NA
HQ
VPC
VPC
VPC
VPC
VPC
VPC
Chicago DX
AP
HQ
London DX
ap-northeast-1
VPC
VPC
VPC
VPC
Transit VPC
VPC
EU
HQ
Tokyo DX

112. Thank you!

113. Remember to complete
your evaluations!