In this session, we walk through the Amazon VPC network presentation and describe the problems we were trying to solve when we created it. Next, we walk through how these problems are traditionally solved, and why those solutions are not scalable, inexpensive, or secure enough for AWS. Finally, we provide an overview of the solution that we've implemented and discuss some of the unique mechanisms that we use to ensure customer isolation, get packets into and out of the network, and support new features like VPC endpoints.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Steve Mueller, Senior Product Manager, VPC, EC2 Networking
November 30, 2016
Another Day, Another Billion Packets
NET401

2. We Have the Cloud
Amazon
EBS
Amazon
RDS
Amazon
ElastiCache
Amazon
Redshift
Amazon EC2 Elastic Load
Balancing

3. Customers Have Datacenters

4. Whiteboard Engineering
Amazon
EBS
Amazon
RDS
Amazon
ElastiCache
Amazon
Redshift
Amazon
EC2
Elastic Load
Balancing

6. EC2 as It Was
10.44.12.4 10.44.12.5
10.44.92.17
10.44.12.27
10.108.6.4

7. Why That Doesn’t Work
192.168.0.0/16
Routing Table
• 192.168.0.0/16: stay here
• 10.44.12.4/32: AWS
• 10.44.92.17/32: AWS
• 10.108.6.4/32: AWS
10.44.0.0/16
10.44.12.4 10.44.12.5
10.44.92.17
10.44.12.27
10.108.6.4

8. Requirements
Customer selected IP addresses
Route aggregation for external connectivity
Conformance with existing network designs

9. 172.31.0.0/18
192.168.0.0/16
Routing Table
• 192.168.0.0/16: stay here
• 172.31.0.0/18: AWS
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.1.9
172.31.2.12
172.31.2.51
Amazon Virtual Private Cloud (Amazon VPC)

10. This Is Just Virtual Networking!
Subnet ~= VLAN
VPC ~= VRF (virtual routing and forwarding)
But…

11. Scaling Challenges
VLAN ID space is constrained
• 12 bits => 4096 total VLANs
VRF support is constrained
• Large routers => 1-2 thousand VRFs
Fixed ratio of VLANs:VRFs

12. Router and Capacity Dimensions
Big Router
Data Plane
Control
Plane
Big Router
Data Plane
Control
Plane

13. An Example
Average router configuration line: 50 chars
Config per VPC: 10 lines
Subnets per VPC: 4
Config per subnet: 5 lines
Total VPCs: 2,000
Config size: 3 MB

14. But…
Doesn’t scale
• 12 bit VLAN ID = 4096 VLANs (not enough)
• BIG routers support 4000 VRFs ($200k+)
Large VLANs make NEs cry
Tied to vendor bugfix cycles (6 months +)
We want commodity, fungible network gear
• BIG virtual routers are built by few companies
• Interoperability of advanced features is marginal

15. Silos of Capacity
A
C
B
FE
D
G
A AA
A
B
C
B B
B B
C
D
F FF
D
D
B
G G
/4 /4
/40 /40
0
0
0
0
1324 132
C
G G
3 27
D DD
9910
F F F F F
1815 40
BB B B B
BB B B B
BB B B B
B B

16. Implementation Requirements
Scale to millions of environments the size of Amazon.com
Any server, anywhere in a region can host an instance
attached to any subnet in any VPC

17. Concepts
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
Server:
Physical host in an
Amazon data center
Instance:
Amazon EC2
instance owned by a
customer
VPC:
Amazon Virtual
Private Cloud
owned by a
customer
VPC ID:
Identifier for a VPC
such as vpc-
1a2b3c4d
Mapping Service:
Distributed lookup
service. Maps VPC
+ Instance IP to
server

18. Layer 2 (L2): Ethernet
10.0.0.2
10.0.0.3
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
The switch floods the
ARP request out all
ports
Ethernet Switch
L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
The switch snoops the
ARP response and
learns the port for
MAC(10.0.0.3).
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…

19. Layer 2 (L2): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
L2 Src: MAC(10.0.0.3)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.3 is at
MAC(10.0.0.3)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.3
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.0.3)
10.0.0.2

20. Server 192.168.0.3
Server 192.168.0.4
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
…
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.1.4
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue10.0.0.2 is at
192.168.0.3
Layer 2 (L2): VPC
…

21. VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
Src: 192.168.0.4
Dst: Mapping Service
Query:
Grey 10.0.0.3
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?

22. VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
Src: 192.168.0.4
Dst: Mapping Service
Query:
Blue 10.0.0.3
L2 Src: MAC(10.0.0.4)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.3?
192.168.0.4 is not
hosting any instances
in VPC Blue.
Mapping Denied
Alarm Raised

23. VPC Isolation
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
10.0.0.2
…
L2 Src: MAC(10.0.0.4)
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.4
L3 Dst: 10.0.0.3
ICMP/TCP/UDP/…
VPC: Blue
Src: 192.168.0.4
Dst: 192.168.1.4
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.4 is at
192.168.0.4
Src: Mapping Service
Dst: 192.168.1.4
Mapping invalid!
192.168.1.4 does not
deliver the packet to
the instance.
Alarm Raised.

24. Layer 3 (L3): IP Routing
10.0.0.2
10.0.1.3
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
Ethernet Switch
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
Router
Ethernet Switch
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

25. Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
L2 Src: MAC(10.0.0.2)
L2 Dst: ff:ff:ff:ff:ff:ff
ARP Who has
10.0.0.1?
L2 Src: MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
ARP 10.0.0.1 is at
MAC(10.0.0.1)
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.0.1
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: Gateway
MAC: MAC(10.0.0.1)
10.0.0.2

26. Layer 3 (L3): VPC
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
10.0.1.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
Src: 192.168.0.3
Dst: Mapping Service
Query:
Blue 10.0.1.3
Src: Mapping Service
Dst: 192.168.0.3
Reply:
Host: 192.168.1.4
MAC: MAC(10.0.1.3)
10.0.0.2
L2 Src: MAC(10.0.0.2)
L2 Dst: MAC(10.0.0.1)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.1.4
Src: 192.168.1.4
Dst: Mapping Service
Validate:
Blue 10.0.0.2 is at
192.168.0.3
Src: Mapping Service
Dst: 192.168.1.4
Mapping valid:
Blue 10.0.0.2 is at
192.168.0.3
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

27. Caching
Server 192.168.0.3
Server 192.168.0.4
…
Server 192.168.1.3
Server 192.168.1.4
…
10.0.0.2
10.0.0.3
10.0.0.4
10.0.0.4
10.0.0.2
10.0.0.5
10.0.0.3
Mapping Service
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

28. 10.0.0.0/18
172.16.0.0/16
10.0.0.0/24 10.0.1.0/24
10.0.0.7
10.0.0.8
10.0.0.9
10.0.1.12
10.0.1.51
VPC: Blue
Src: 192.168.0.3
Dst: ???
L3 Src: 10.0.0.7
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Getting Home – or Anywhere, Really

29. Edges
Server 192.168.0.3
Server 192.168.0.4
…
Edge 192.168.4.3
Edge 192.168.4.4
10.0.1.3
10.0.0.4
10.0.0.2
Mapping Service
10.0.0.2
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Host 10.0.0.4  192.168.0.4
Host 10.0.1.4  192.168.0.4
…
172.16.0.0/16  Edge 192.168.4.3
…

30. Edges: VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
VPN

31. Edges: Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
AWS
Direct Connect

32. Edges: Internet (IGW)
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
Internet
54.148.157.46

33. Edges: Recap
VPN
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
IPSEC Stuff
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
AWS Direct Connect
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
802.1Q VLAN Tag
Src: 54.68.100.245
Dst: 205.251.242.54
L3 Src: 10.0.0.2
L3 Dst: 172.16.14.17
ICMP/TCP/UDP/…
Internet
Edge 192.168.4.3
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.3
L3 Src: 10.0.0.2
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…
L3 Src: 54.148.157.46
L3 Dst: 176.32.96.190
ICMP/TCP/UDP/…

34. Image credit: Wikipedia
https://en.wikipedia.org/wiki/1918_Eighth_Avenue
A Brief Diversion

35. VPC Pricing
Cost per VPC: $0.00
Cost per subnet: $0.00
Upcharge per instance: $0.00

36. Nov 10, 2010

37. 172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.2.12
172.31.2.51
VPC as a Platform

38. VPC as a Platform
VPN and Direct Connect
Security group egress filtering
Network ACLs
Routing tables
Elastic Network Interfaces (ENIs)
Multiple IPs

39. Amazon S3 Endpoints
172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7 172.31.2.12

40. Amazon S3 Endpoints
172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7 172.31.2.12

41. Server 192.168.0.3
Server 192.168.0.4
…
Edge 192.168.4.3
Edge 192.168.4.4
10.0.1.3
10.0.0.4
10.0.0.2
10.0.0.2
L3 Src: 10.0.0.2
L3 Dst: 54.231.33.89
TCP/HTTP/…
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.4
L3 Src: 10.0.0.2
L3 Dst: 54.231.33.89
TCP/HTTP/…
Edges Mapping Service
Host 10.0.0.4  192.168.0.4
Host 10.0.1.4  192.168.0.4
…
172.16.0.0/16  Edge 192.168.4.3
S3.us-east-1  Edge 192.168.4.4
…

42. A New Edge: S3
Edge 192.168.4.4
VPC: Blue
Src: 192.168.0.3
Dst: 192.168.4.4
L3 Src: 10.0.0.2
L3 Dst: 54.231.33.89
TCP/HTTP/…
VPC Endpoint 1a2b3c4d
Src: 54.68.100.245
Dst: 54.231.33.89
L3 Src: 10.0.0.2
L3 Dst: 54.231.33.89
TCP/HTTP/…
S3 endpoint

43. S3 Endpoints and Policy
172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7 172.31.2.12
{
"Statement": [
{
"Sid": "Access-to-specific-bucket-only",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::my_secure_bucket",
"arn:aws:s3:::my_secure_bucket/*"]
}
]
}
{
"Statement": [
{
"Sid": "Access-to-specific-VPC-only",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::my_secure_bucket",
"arn:aws:s3:::my_secure_bucket/*"],
"Condition": {
"StringNotEquals": {
"aws:sourceVpc": "vpc-111bbb22"
}
}
}
]
}

44. Simple Complex
Limited Flexible
EC2 VPC

45. 172.31.0.0/18
172.31.1.0/24 172.31.2.0/24
172.31.1.7
172.31.1.8
172.31.1.9
172.31.2.12
172.31.2.51
Default VPC

46. Simple Complex
Limited Flexible
EC2 - VPC

47. Related Sessions
NET201 - Creating Your Virtual Data Center: VPC Fundamentals and Connectivity Options
NET303 - NextGen Networking: New Capabilities for Amazon’s Virtual Private Cloud
NET304 - Moving Mountains: Netflix's Migration into VPC
NET402 - Deep Dive: AWS Direct Connect and VPNs
NET404 - Making Every Packet Countr

48. Remember to complete
your evaluations!

49. Thank you!