More Related Content Similar to Container Networking Deep Dive with Amazon ECS - CON401 - re:Invent 2017 (20) More from Amazon Web Services (20) Container Networking Deep Dive with Amazon ECS - CON401 - re:Invent 20171. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Networking Deep
Dive with Amazon ECS
D e e p a k D a y a m a , S r . P r o d u c t M a n a g e r — A m a z o n E C S
S h a k e e l S o r a t h i a , V P E n g i n e e r i n g , F o x D C G
N o v e m b e r 3 0 , 2 0 1 7
C O N 4 0 1
2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Container Networking with Amazon ECS
• Bridge Mode
• AWS VPC Mode
Networking for Microservices @Fox DCG
3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terminology
4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Elastic Network Interface (ENI)
“An ENI is a virtual network interface that
you can attach to an instance* in a VPC”
An ENI can have following attributes:
• A primary private IPv4 address
• One or more secondary private IPv4 addresses
• One public IPv4 address
• One or more IPv6 addresses
• One or more security groups
• A MAC address
• A source/destination check flag
• A description
* We will revisit this later in the presentation
5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Define application containers: Image
URL, CPU & Memory requirements,
etc.
register
Task Definition
create
Cluster
• Infrastructure Isolation
boundary
• IAM Permissions boundary
run
Task
• A running instantiation of
a task definition
create
Service
Application
Load Balancer
• Maintain n running copies
• Integrated with ALB
• Unhealthy tasks
automatically replaced
ECS CONSTRUCTS
6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Definition
Example
• Embodiment of the application
requirements
• Defines containers, their image source,
logging preferences, and its
networking configuration
7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Networking Modes with ECS
8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Networking Modes with ECS
Host mode
Bridge mode
Task Networking (awsvpc) mode (New!)
None
9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Networking Modes with ECS
Host mode
Bridge mode
Task Networking (awsvpc) mode (New!)
None
10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bridge Mode
11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bridge Mode—Quick Primer
Containers share the same network interface as the instance
Each container gets a private IP and uses the Docker bridge for any communication.
Multiple Tasks use the same ENI
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overview of Bridge Networking Mode
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 1
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 2
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 3
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 4
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 5
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 6
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 1
Container 2
172.16.0.0
172.16.1.0
172.16.2.0
VPC
10.0.0.27/24
docker0
Io
eth0 172.17.0.1/16
Io
Io
ve-c2
(172.17.0.3/16)
ve-c1
(172.17.0.2/16)
Default/Root Global Namespace
Container 3
Container 4
172.16.0.0
172.16.1.0
172.16.2.0
10.0.0.26/24
docker0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
172.16.0.0
172.16.1.0
172.16.2.0
19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Working with Load Balancers
ECS Instance
172.31.2.165:8080
ENI
Private IP
172.31.2.165
172.31.2.165:80
Internet
ECS Instance
172.31.2.164:8080
ENI
Private IP
172.31.2.164
172.31.2.164:80
Containers
register with the
load balancer on
different ports
Security group
rules set to allow
ports 80 and 8080
20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Working with Load Balancers
Q. What if multiple applications
were listening on the same port?
Use dynamic port mapping
21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS Instance
80
ENI
Private IP
172.31.2.165
80
Internet
ECS Instance
80
ENI
Private IP
172.31.2.164
80172.31.2.164:6000
172.31.2.164:5000
172.31.2.165:1234
172.31.2.165:2345
22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges
Performance
Lack of finer grained access control policies
No routable IP addresses for containers
23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWSVPC Mode
New!!
24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking for ECS
https://github.com/aws/amazon-ecs-agent/blob/master/proposals/eni.md
25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWSVPC Mode for Fargate
26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Configuration
- We create an ENI on your behalf
- ENI gets a private IP from the subnet
- Security group allows local traffic only
aws register-task-definition
{
“family": "helloFargate",
"networkMode":"awsvpc"
"cpu": 1024,
"memory": 512,
"containerDefinitions": [ ... ],
...
}
aws run-task
-- launch-type FARGATE
-- network-configuration
“awsVpcConfiguration = {
subnets=[subnet-id],
securityGroups=[sg-id]
}”
...
27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking – Task Definition
28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking – RunTask
29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking – ENI attachment
Default/Root Global Namespace
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
1. Pre ENI Attachment: The
Primary ENI (eth0) is in the
default namespace
2. ENI Attachment: The new
ENI (eth1) is in the default
namespace
Default/Root Global Namespace
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0eth1
ECS Instance ECS Instance
30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking – ENI provisioning
Default/Root Global Namespace
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
1. Pre ENI Attachment: The
Primary ENI (eth0) is in the
default namespace
2. ENI Attachment: The new
ENI (eth1) is in the default
namespace
3. ENI Provisioned: The ECS Agent
invokes CNI plugins to move the
new ENI into a new namespace and
configure it with addresses and routes
Default/Root Global Namespace
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0eth1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Task Namespace
ECS Instance ECS Instance ECS Instance
31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ENI Provisioning Workflow
AWS CLI Run Task Find Resources
(i.e., find the instance
to place it on)
Create ENI Attach ENI Task=RUNNING
32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ENI De-Provisioning Workflow
AWS CLI Stop Task Detach ENI Destroy ENI Terminate Task
33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Task Networking – Attachment Details
34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
10.0.0.28/24
10.0.0.29/24
10.0.0.26/24
10.0.0.27/24
Task 2Task 1
AWS VPC
35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 2
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
10.0.0.28/24
10.0.0.29/24
10.0.0.26/24
10.0.0.27/24
Task 2Task 1
AWS VPC
36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 3
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
10.0.0.28/24
10.0.0.29/24
10.0.0.26/24
10.0.0.27/24
Task 2Task 1
AWS VPC
37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inter-Container Communication 4
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
Default/Root Global Namespace
ecs0
Io
eth0
172.16.0.0
172.16.1.0
172.16.2.0
docker0
Io
eth1
172.16.0.0
172.16.1.0
172.16.2.0
ve-c1
10.0.0.28/24
10.0.0.29/24
10.0.0.26/24
10.0.0.27/24
Task 2Task 1
AWS VPC
38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing Between Bridge
and Task Networking Modes
Consider using AWS VPC mode for
new services that are attached to ALB/NLB
• Migration from bridge to task networking mode (coming soon!)
Amazon EC2 ENI limits
• Fargate is an alternative
40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Running ECS
with a Service Mesh
Shakeel Sorathia, VP Engineering
F o x D C G
41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
21st Century Fox is home to a global portfolio
of cable and broadcasting networks and properties,
including FOX, FX, FXX, FXM, FS1, Fox News Channel,
Fox Business Network, FOX Sports, Fox Sports
Network, National Geographic, STAR India, 28 local
television stations in the U.S., and more than 350
international channels; film studio Twentieth
Century Fox Film; and television production studios
Twentieth Century Fox Television and a 50%
ownership interest in Endemol Shine Group.
The Digital Consumer Group is an organization inside
the company tasked with the digital distribution of
the companies’ content to consumers.
About Fox and the
Digital Consumer Group
42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Deliver
All APIs powering our experiences are
running on AWS
API’s delivered through Amazon API Gateway
Microservice architecture with
services written in Node.js and Go
Different teams deploy services differently
All services are delivered through
Docker containers running on ECS
Amazon API
Gateway
Amazon ECS
43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Problems That
Needed to Be Solved
Microservice architecture with lots
of services, utilizing bridge mode
networking, across 100s of instances
Teams needed to be able to determine
how to route an individual service
between blue/green deployments
Teams needed to be able to route a single
request to a specific service for testing
Maintain multiple versions of the same
service
Be very ephemeral and deal with all kinds
of failures
44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Investigated Solutions
Utilizing ALBs in front of all microservices
• Target groups have a limit of 1000 targets
• This led to 100s of ALBs, plus API Gateway
integrations were very complex
Direct service-to-service
communication through DNS
• Very complex DNS structure, especially
if trying to do blue/green deployments
Utilizing software to create a service mesh
• This is ultimately where we ended up
45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is a Service Mesh?
A dedicated infrastructure layer for
making service-to-service communication
safe, fast, and reliable
Utilizing some form of service discovery allows
the service mesh software to route requests
to a healthy instance of a specific service
The service mesh software that
we elected to utilize is linkerd
46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Does the Implementation Look Like?
Amazon API Gateway
NLB
Internet
Communication secured through client
certification such that only API Gateway
can talk to linker thru NGINX
NGINX layer used for
Client Cert Validation
While they look
different here,
they could be
the same set of
physical servers
Amazon ECS
Amazon ECS FOX DCG Microservices
Auto Scaling
47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service-to-Service Communication
Service containers are
deployed through ECS and
use Bridge networking
All service requests go
to the local linkerd
linkerd is deployed to every ECS Container Instance linkerd finds a service instance and proxies request
ECS Container Instance ECS Container Instance
Service B Service A
linkerd linkerd
Private Subnet
AZ
VPC
Region
VPC
48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Implementation Details
All ECS Instances reside in private subnet
Service containers are deployed through ECS
Container instances utilize the ECS-optimized AMI
Linkerd is added to all ECS Container
Instances at startup through userdata
Services send all service-service
requests to their local linkerd instance
All services have a name that
other services can use to talk to it
Linkerd uses this name to find
that service’s routing in its delegation
/serviceA_v1 => .95 * /blue/
serviceA_v1 & .05 * /green/serviceA_v1;
Linkerd decides how to route it and then
finds an available instance and proxies the request
ECS Instances are horizontally scalable and use EC2 Auto Scaling
groups to scale out when available memory or CPU is running low
49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What We Achieved
Gave teams the ability to instantly
change routing for a microservice.
• By modifying the delegation tab for a service,
a team can decide where the traffic should go.
Linkerd has some resiliency functions built in.
For example, it can retry idempotent requests.
Because it ties in with a service discovery
backend, it can immediately find healthy
containers to route to.
Another nice feature of linkerd is to be able
to determine slower instances and route fewer
requests to these instances. Helps to deal
with the noisy neighbor issue.
50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges We Plan to Tackle
Security around specific
services is still a challenge
• Task IAM roles effective for resource access
• Using Task ENIs would give us the ability to use
Security Groups on containers. (Big win for PII or
PCI data)
Linkerd doesn’t natively understand
how to find the closest container
(though latency routing helps with this)
51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!
dayamad@amazon.com
@saysdd