An overview of how Netflix integrates AWS VPC networking into Titus, its container cloud.

1. ●
○
○
●
○
○
○
1

2. ●
●
○
○
●
○
●
○
2

3. ●
○
○
○
○
●
●
○
■
3

4. ●
○
●
○
○
■
■
4

5. ●
○
○
●
○
■
○
○
5

6. ●
●
●
●
●
○
●
6

7. 7
Titus
mesos
executor
Titus network
driver
Docker engine
EC2 Instance
In: Network params
Out: Network pod root
New task
Container create/start
--net=container:<pod id>
Task Status
create/start pod
container

8. ●
○
●
○
●
○
○
●
8

9. 9
Create
NS Configurator
IP Allocator
NS Allocator
Http
IP +
params
NS ref
Configured
network ns
(pod root)
params
Container id

10. ●
●
●
●
●
○
●
●
○
10

11. EC2 Instanceeth1
ENI1
SecGrp=A
eth2
ENI2
SecGrp=X
eth3
ENI3
SecGrp=Y,Z
IP 2 (primary)
IP 3
IP 6 (primary)IP 1 (primary)
IP 4
IP 5
IP 7
IP 8

12. 12

13. ●
○
●
●
○
●
●
○
13

14. ●
○
●
●
●
○
○
●
14

15. ●
○
●
○
●
○
15

16. ●
○
●
●
○
●
○
●
16

17. No IP, SecGrp A
Task 0
SecGrp Y,Z
Task 1 Task 2 Task 3
Titus EC2 Host VMeth1
ENI1
SecGrp=A
eth2
ENI2
SecGrp=X
eth3
ENI3
SecGrp=Y,Z
IP 1
IP 2
IP 3
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
appapp
veth<id>
Linux Policy Based
Routing + Traffic Control
Titus
EC2
Metadata
Proxy
169.254.169.254
IPTables NAT (*)
* **
169.254.169.254
Non-routable IP
*

18. ●
○
○
●
18

19. ●
● <IP>/32
○ via eth0
●
●
19

20. ● Container IP: 100.66.23.19
● Container Device: vethA
● Eni IP: 100.66.30.31/20
● Eni GW: 100.66.16.1
● Eni Device: eth1
● Routing tables:
○ tocontainer, fromcontainer
20

21. # ip addr show eth0
eth0: … mtu 1500 qdisc tbf state UP group default
inet 100.66.23.19/32 ...
# ip route show
default via 100.66.30.31 dev eth0
100.66.30.31 dev eth0 scope link
21

22. # ip route show | grep eth1
100.66.16.0/20 dev eth1 proto kernel scope link src 100.66.30.31
# ip rule show | grep 100.66.23.19
from all to 100.66.23.19 iif eth1 lookup tocontainer
from 100.66.23.19 iif vethA lookup fromcontainer
# ip route show table tocontainer | grep 100.66.23.19
100.66.23.19 dev vethA scope link
# ip route show table fromcontainer
default via 100.66.16.1 dev eth1
22

23. ●
●
●
●
23

24. ●
●
○
●
○
●
24

25. ●
○
○
○
○
○
●
○
25

26. 26

27. 27

28. 28

29. ●
●
●
○
●
○
○
29