In this talk, we walk through the VPC network presentation, and describe the problems we were trying to solve. Next, we walk through how these problems are traditionally solved, and why those solutions are not scalable, cheap, or secure enough for AWS. Finally, we provide an overview of the solution that we've implemented and discuss some of the unique mechanisms that we use to ensure customer isolation.

1. A Day in the Life of a Billion Packets
Eric Brandwine, AWS Security
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. We have the cloud
EC2
RDS
Elastic Load
Balancing
EBS
Redshift
ElastiCache
AWS Cloud

3. Customers have Data centers

4. Whiteboard Engineering
EC2
RDS
Elastic Load
Balancing
EBS
Redshift
ElastiCache
AWS Cloud

6. EC2 as it was
10.44.12.5
10.44.12.4
10.44.92.17
10.44.12.27
10.108.6.4
Amazon EC2

7. Why that doesn’t work
192.168.0.0/16
10.44.0.0/16
10.44.12.5
10.44.12.4
10.44.92.17
Routing Table
• 192.168.0.0/16:
• 10.44.12.4/32:
• 10.44.92.17/32:
• 10.108.6.4/32:
10.44.12.27
stay here
AWS
AWS
AWS
10.108.6.4
Amazon EC2

8. Requirements
• Customer Selected IP Addresses
• Route Aggregation for External Connectivity
• Conformance with Existing Network Designs

9. Virtual Private Cloud
192.168.0.0/16
172.31.1.7
172.31.1.8
Routing Table
• 192.168.0.0/16:
• 172.31.0.0/18:
stay here
AWS
172.31.2.12
172.31.2.51
172.31.1.9
172.31.1.0/24
172.31.0.0/18
172.31.2.0/24

10. This is just virtual networking!
• Subnet ~= VLAN
• VPC ~= VRF (Virtual Routing and Forwarding)
• But…

11. Scaling Challenges
• VLAN ID space is constrained
– 12 bits => 4096 total VLANs
• VRF support is constrained
– Large routers => 1-2 thousand VRFs
• Fixed ratio of VLANs:VRFs

12. Router and capacity dimensions
Big Router
Big Router
Control
Plane
Control
Plane
Data Plane
Data Plane

13. An Example
•
•
•
•
•
•
Average Router Configuration Line:
Config per VPC:
Subnets per VPC:
Config per Subnet:
Total VPCs:
Config size:
50 chars
10 lines
4
5 lines
2,000
3MB

14. Silos of Capacity
2
3
1
0 /4
4
2
3
1
0 /4
A
A
C
C
B
B
D
D
D
C
C
A
A
E
D
D
D
D
F
G
G
G
G
G
F
F
F
F
F
F
15
10/40
9
7
3
0
A
F
F
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
40
18 /40
9
2
0

15. Implementation Requirements
• Scale to millions of environments the size of
Amazon.com
• Any server, anywhere in a region can host an
instance attached to any Subnet in any VPC

16. Concepts
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.0.4
10.0.0.5
10.0.0.3
…
…
Mapping
VPC
VPC:ID:
Instance:
Server: Service:
Distributed lookup
Identifier host VPC
Amazon Virtual
Physical EC2 in an
for a
service. Maps
such as owned
Private Cloud by
instance datacentera
Amazon vpc- VPC
+ Instance
1a2b3c4d
owned by
customer aIP to
server
customer

17. L2 - Ethernet
Ethernet Switch
10.0.0.2
10.0.0.3
The switch floods the
L2 Src: MAC(10.0.0.2)
MAC(10.0.0.3)
snoops the
ARP request out all
L2 Dst: MAC(10.0.0.2)
response and
MAC(10.0.0.3)
ff:ff:ff:ff:ff:ff
L3 Src:
learns
ports the port for
10.0.0.2
L3 Dst:
MAC(10.0.0.3).
ARP Who hasis at
10.0.0.310.0.0.3
MAC(10.0.0.3)
10.0.0.3?
ICMP/TCP/UDP/…

18. L2 - VPC
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.0.3
10.0.0.4
Src: 192.168.0.3
L2 Src: MAC(10.0.0.2)
Mapping Service
MAC(10.0.0.3)
Dst: Mapping Service
L2 Dst: MAC(10.0.0.2)
192.168.0.3
ff:ff:ff:ff:ff:ff
10.0.0.5
…
Reply:
Query:
ARP Who hasis at
10.0.0.3
MAC(10.0.0.3)
10.0.0.3?10.0.0.3
Host: 192.168.1.4
Orange
MAC: MAC(10.0.0.3)

19. L2 - VPC
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.3
10.0.0.2
…
Src: 192.168.0.3
Dst: 192.168.1.4
10.0.0.2
Server 192.168.0.4
VPC: Orange
10.0.0.4
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.5
…
Src: 192.168.1.4
Mapping Service
L2 Src: MAC(10.0.0.2)
Dst: Mapping Service
192.168.1.4
L2 Dst: MAC(10.0.0.3)
L3 Src: 10.0.0.2
Mapping
Validate: valid:
L3 Dst: 10.0.0.3
Orange 10.0.0.2 is at
192.168.0.3
ICMP/TCP/UDP/…

20. VPC Isolation
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.0.3
10.0.0.4
L2 Src: MAC(10.0.0.4)
Src: 192.168.0.4
L2 Dst: ff:ff:ff:ff:ff:ff
Dst: Mapping Service
10.0.0.5
…
ARP Who has
Query:
10.0.0.3?
Grey 10.0.0.3

21. VPC Isolation
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.0.3
10.0.0.4
10.0.0.5
…
192.168.0.4 is not
L2 Src: MAC(10.0.0.4)
Src: 192.168.0.4
hosting any instances
L2 Dst: ff:ff:ff:ff:ff:ff
Dst: Mapping Service
in VPC Orange.
ARP Who has
Query:
Mapping 10.0.0.3
10.0.0.3?Denied
Orange
Alarm Raised

22. VPC Isolation
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.3
10.0.0.2
…
Src: 192.168.0.4
Dst: 192.168.1.4
10.0.0.2
Server 192.168.0.4
VPC: Orange
10.0.0.4
Server 192.168.1.4
10.0.0.3
10.0.0.4
10.0.0.5
…
192.168.1.4 does
Src: 192.168.1.4 not
Mapping Service
L2 Src: MAC(10.0.0.4)
deliver the packet to
Dst: Mapping Service
192.168.1.4
L2 Dst: MAC(10.0.0.3)
theSrc: 10.0.0.4
L3 instance.
Mapping
Validate: invalid!
L3 Dst: 10.0.0.3
Alarm Raised.
Orange 10.0.0.4 is at
192.168.0.4
ICMP/TCP/UDP/…

23. L3 – IP Routing
Ethernet Switch
Router
10.0.0.2
Ethernet Switch
10.0.1.3
L2 Src: MAC(10.0.0.2)
MAC(10.0.0.1)
L2 Dst: MAC(10.0.0.2)
MAC(10.0.0.1)
ff:ff:ff:ff:ff:ff
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ARP Who hasis at
10.0.0.1
MAC(10.0.0.1)
10.0.0.1?
ICMP/TCP/UDP/…
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3
ICMP/TCP/UDP/…

24. L3 - VPC
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.1.3
10.0.0.4
Src: 192.168.0.3
L2 Src: MAC(10.0.0.2)
Mapping Service
MAC(10.0.0.1)
Dst: Mapping Service
L2 Dst: MAC(10.0.0.2)
192.168.0.3
ff:ff:ff:ff:ff:ff
10.0.0.5
…
Reply:
Query:
ARP Who hasis at
10.0.0.1
MAC(10.0.0.1)
10.0.0.1?10.0.0.1
Host: Gateway
Orange
MAC: MAC(10.0.0.1)

25. L3 - VPC
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.3
10.0.0.2
10.0.0.2
Src: 192.168.0.3
Dst: 192.168.1.4
Server 192.168.0.4
10.0.0.4
10.0.0.5
10.0.0.4
Server 192.168.1.4
VPC: Orange
…
L2 Src: MAC(10.0.0.2)
Src: 192.168.0.3
192.168.1.4
Mapping Service
MAC(10.0.1.1)
L2 Dst: MAC(10.0.0.1)
Dst: Mapping Service
192.168.1.4
192.168.0.3
MAC(10.0.1.3)
L3 Src: 10.0.0.2
Mapping
Validate:
L3 Dst:
Reply: valid:
Query: 10.0.1.3
Host: 192.168.1.4
Orange 10.0.1.3 is at
10.0.0.2
ICMP/TCP/UDP/…
192.168.0.3
MAC: MAC(10.0.1.3)
10.0.1.3

26. Caching
Mapping Service
Server 192.168.1.3
Server 192.168.0.3
10.0.0.2
10.0.0.3
10.0.0.2
10.0.0.4
Server 192.168.1.4
Server 192.168.0.4
10.0.0.3
10.0.0.4
10.0.0.5
ICMP/TCP/UDP/…
…
…
L2 Src: MAC(10.0.1.1)
L2 Dst: MAC(10.0.1.3)
L3 Src: 10.0.0.2
L3 Dst: 10.0.1.3

27. VPC Pricing
Cost per VPC:
Cost per Subnet:
Upcharge per Instance:
$0.00
$0.00
$0.00

28. Nov 10, 2010

29. VPC as a Platform
172.31.1.7
172.31.2.12
172.31.1.8
172.31.2.51
172.31.1.0/24
172.31.2.0/24
172.31.0.0/18

30. VPC as a Platform
•
•
•
•
•
•
VPN and Direct Connect
Security Group Egress Filtering
Network ACLs
Routing Tables
Elastic Network Interfaces (ENIs)
Multiple IPs

31. EC2
VPC
Simple
Complex
Limited
Flexible

32. Default VPC
172.31.1.7
172.31.2.12
172.31.1.8
172.31.2.51
172.31.1.9
172.31.1.0/24
172.31.0.0/18
172.31.2.0/24

33. EC2 - VPC
Simple
Complex
Limited
Flexible

34. Other VPC Sessions
ARC202: High Availability Application
Architectures in Amazon VPC
ARC401: From One to Many: Evolving VPC
Design
CPN208: Selecting the Best VPC Network
Architecture (single VPC vs. multiple VPCs)
CPN301: Amazon EC2 to Amazon VPC: A case
study (this is the migration story)

35. Please give us your feedback on this
presentation
CPN401
As a thank you, we will select prize
winners daily for completed surveys!