AWS EC2 Container Service and Registry services deep dive

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apr 21 2016
Amazon EC2 Container
Service Deep Dive
Shiva N, Solution Architect, AWS

2. Agenda
The Basics
Infrastructure Setup
Infrastructure Management
Deploying Applications
PaaS on ECS
Using the CLI

3. TaskDefinitions
Containers
Clusters
Container Instances
Key Components

4. Amazon ECS Infrastructure
Setup

5. Amazon ECS Infrastructure Setup
Amazon ECS Cluster Setup
Amazon ECR Setup

6. Amazon ECS Cluster Setup

7. Amazon ECS Cluster Setup
There are many ways to provision cluster infrastructure
v AWS – CloudFormation, Simple Systems Manager,
Autoscale Groups, OpsWorks, ECS-CLI
v Others - Terraform, PaaS, Partners
Let’s talk about CloudFormation

8. Cluster Setup with AWS CloudFormation
CloudFormation supports ECS cluster, service and task
definition resources
Use AWS::IAM::Role to create ECS service role and
container instances role
Launch container instances using
AWS:AutoScaling::LaunchConfiguation and
AWS:AutoScaling::AutoScalingGroup

9. Cluster Setup with AWS CloudFormation
"Resources" : {
"ECSCluster": {
"Type": "AWS::ECS::Cluster"
},
"ECSAutoScalingGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"VPCZoneIdentifier" : { "Ref" : "SubnetID" },
"LaunchConfigurationName" : { "Ref" : "ContainerInstances" },
"MinSize" : "1",
"MaxSize" : { "Ref" : "MaxSize" },
"DesiredCapacity" : { "Ref" : "DesiredCapacity" }
},
[…]
},

10. Cluster Setup with AWS CloudFormation
"ContainerInstances": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"commands" : {
"01_add_instance_to_cluster" : {
"command" : { "Fn::Join": [ "", [ "#!/bin/bashn", "echo
ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] }
}
},
[…]
}
}
}

11. Amazon ECR Setup

12. Amazon ECR Setup
You have read and write access to the repositories you
create in your default registry, i.e.
<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com
Repository names can support namespaces, e.g. team-
a/web-app.
Repositories can be controlled with both IAM user access
policies and repository policies.

13. Amazon ECR Setup
# Authenticate Docker to your Amazon ECR registry
> aws ecr get-login
docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
# Create a repository called ecr-demo
> aws ecr create-repository --repository-name ecr-demo
# Push an image to your repository
> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

14. Amazon ECS Infrastructure
Management

15. Amazon ECS Infrastructure Management
Monitoring & Logging
Scaling ECS
Service Discovery & Configuration Management
Security

16. Monitoring & Logging

17. Monitoring and Logging on Amazon ECS
Monitoring with Amazon CloudWatch
Configuring logging in Task Definition
Amazon CloudTrail
Monitoring Amazon ECS with Datadog
Monitoring Amazon ECS with Sysdig Cloud

18. Monitoring with Amazon CloudWatch
Metric data sent to CloudWatch in 1-minute periods and
recorded for a period of two weeks
Available metrics: CPUReservation, MemoryReservation,
CPUUtilization, MemoryUtilization
Available dimensions: ClusterName, ServiceName

19. Monitoring with Amazon CloudWatch

20. Monitoring with Amazon CloudWatch
Use the Amazon CloudWatch Monitoring Scripts to monitor
additional metrics, e.g. disk space:
# Edit crontab
> crontab -e
# Add command to report disk space utilization to CloudWatch every five minutes
*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-
space-avail --disk-path=/ --from-cron

21. Configuring Logging in Task Definition
logConfiguration task definition parameter
Requires version 1.18 or greater of the Docker Remote API
Maps to docker run --log-driver option
Log drivers: json-file, syslog, journald, gelf, fluentd

22. Logging with Amazon CloudWatch Logs
• Logging container with syslogd
and CloudWatch Logs Agent
• Attach /var/log Volume to
Logging container (Sidecar
pattern)
• Link other containers
syslogd
CloudWatch Logs
Agent
CloudWatch
Logs
Container instance
ECS Cluster
ECS Agent
Logs
Docker
Logs

23. Logging Amazon ECS API with AWS CloudTrail
{
"eventVersion": "1.03",
"userIdentity": {…},
"eventTime": "2015-10-12T13:57:33Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "CreateCluster",
"awsRegion": "eu-west-1",
"sourceIPAddress": "54.240.197.227",
"userAgent": "console.amazonaws.com",
"requestParameters": {
"clusterName": "ecs-cli"
},

24. Logging Amazon ECS API with AWS CloudTrail
"responseElements": {
"cluster": {
"clusterArn": "arn:aws:ecs:eu-west-
1:560846014933:cluster/ecs-cli",
"pendingTasksCount": 0,
"registeredContainerInstancesCount": 0,
"status": "ACTIVE",
"runningTasksCount": 0,
"clusterName": "ecs-cli",
"activeServicesCount": 0
}
},
[…]

25. Monitoring Amazon ECS with Datadog

26. Monitoring Amazon ECS with Sysdig Cloud

27. Scaling Amazon ECS

28. Scaling Amazon ECS
AutoScaling your Amazon ECS cluster
Scaling your Services with Lambda

29. Setup ECS Cluster with AutoScaling
Create LaunchConfiguration
• Pick instance type
depending on resource
requirements, e.g. memory
or CPU
• Use latest Amazon Linux
ECS-optimized AMI, other
distros available
Create AutoScaling group and
set to cluster initial size

30. AutoScaling your Amazon ECS Cluster
• Create CloudWatch
alarm on a metric, e.g.
MemoryReservation
• Configure scaling
policies to increase and
decrease the size of your
cluster

31. Scaling your Services with Lambda
• Cloudwatch metrics tied
to SNS
• SNS triggers Lambda
Container Scaling
function
• Lambda scales task
count on cluster
• Bonus - Extensible
‘cluster intelligence’ layer

32. Service Discovery &
Configuration Management

33. Service Discovery on Amazon ECS
Service Discovery with ECS Services & Route 53
Service Discovery with Weaveworks
Service Discovery and Configuration Management with
Consul
Service Discovery and Configuration Management with
etcd

34. Service Discovery with ECS Services & Route 53
• Route 53 private hosted zone
• Set search path on hosts with DHCP option sets
• Define ECS services with ELB
• Create CNAMEs for each ELB

35. Service Discovery with ECS Services & Route 53
Task
Task TaskTask
ECS
Service
Application
router, e.g.
nginx
Internal ELB with
CNAME, e.g.
api.example.com
Route 53
private zone,
e.g.
example.com

36. Service Discovery with Weaveworks
DNS interface for cross-host
container communication
Gossip protocol to share
grouped updates
Overlay network between hosts

37. Service Discovery and Configuration
Management with Consul
Three main components:
• Consul agent - Runs on each node, responsible for
checking the health of the services and of the node
itself.
• One or more Consul servers - Store and replicate
data, leader elected using the Raft consensus
algorithm
• Registrator agent - Automatically
register/deregisters services based on published
ports and metadata from the container environment
variables defined in the ECS task definition

38. Service Discovery and Configuration
Management with Consul
ECSCluster
consul-server
ECS Instance
consul-agent
registrator
ECS Instance
Back end 1
Back end 2
consul-agent
registrator
ECS Instance
Front end
ECSCluster

39. Service Discovery and Configuration
Management with etcd
etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd

40. Security

41. Security
ECS IAM Policies and Roles
ECR IAM Policies and Roles
Image Vulnerability Scanning with Twistlock

42. ECS IAM Policies and Roles
The ECS agent calls the ECS APIs on your behalf, so
container instances require an IAM policy and role that
allows these calls.
The ECS service scheduler calls the EC2 and ELB APIs on
your behalf to register and deregister container instances
with your load balancers.
Use AmazonEC2ContainerServiceforEC2Role and
AmazonEC2ContainerServiceRole managed policies
(respectively)

43. ECR IAM Policies and Roles
ECR uses resource-based permissions to control access.
By default, only the repository owner has access to a
repository.
You can apply a policy document that allows others to
access your repository.
Use managed policies for IAM users or roles that allow
differing levels of control:
AmazonEC2ContainerRegistryFullAccess,
AmazonEC2ContainerRegistryPowerUser or
AmazonEC2ContainerRegistryReadOnly

44. Image Vulnerability Scanning with Twistlock

45. Deploying Applications

46. Deploying Applications
Scheduling Containers
Automating Deployments

47. Scheduling Containers

48. Scheduling Containers on ECS
Batch Jobs
ECS Task scheduler
Run tasks once
Batch jobs
RunTask (random)
StartTask (placed)
Long-Running Apps
ECS Service scheduler
Health management
Scale-up and scale-down
AZ aware
Grouped Containers

49. Scheduling Containers: Long-running App
Optionally run your service behind a load balancer.
One load balancer per service.
ELB currently supports a fixed relationship between the
load balancer port and the container instance port.
If a task fails the ELB health check, the task is killed and
restarted (until service reaches desired capacity).

50. Scheduling Containers: Long-running App
Update service’s task definition (rolling update)
Specify a deployment configuration for your service:
• minimumHealthyPercent: lower limit (as a percentage of
the service's desiredCount) of the number of running
tasks that must remain running in a service during a
deployment.
• maximumPercent: upper limit (as a percentage of the
service's desiredCount) of the number of running tasks
that can be running in a service during a deployment.

51. Scheduling Containers: Long-running app
Deploy using the least space: minimumHealthyPercent =
50%, maximumPercent = 100%

52. Scheduling Containers: Long-running App
Deploy quickly without reducing service capacity:
minimumHealthyPercent = 100%, maximumPercent =
200%

53. Scheduling Containers: Long-running App
Blue-Green Deployments
• Define two ECS services
• Each service is associated w/ ELB
• Both ELBs in Route 53 record set
with weighted routing policy, 100%
Primary, 0% Secondary
• Deploy to Blue or Green service and
switch weights
TaskTask
Route 53
record set
with
weighted
routing
policy
0%
100%

54. Automating Deployments

55. Automating Deployments
Continuous Delivery to ECS with Jenkins
Continuous Delivery to ECS with Shippable

56. Continuous Delivery to ECS with Jenkins
4. Push image to
Docker registry
2. Build image
from sources
3. Run test on image
1. Code push
triggers build
5. Update Service
6. Pull image

57. Continuous Delivery to ECS with Jenkins
Easy Deployment
Developers – Merge into master, done!
Jenkins Build Steps
Trigger via Webhooks, Monitoring, Lambda
Build Docker image via Build and Publish plugin
Push Docker image into Registry
Register Updated Job with ECS API

58. Continuous Delivery to ECS with Shippable

59. ECS CI/CD Partners

60. PaaS on ECS

61. PaaS on ECS
AWS Elastic Beanstalk
Convox
Remind Empire

62. AWS Elastic Beanstalk
Uses Amazon ECS to coordinate deployments to
multicontainer Docker environments
Takes care of tasks including cluster creation, task
definition and execution

63. AWS Elastic Beanstalk
Elastic Beanstalk uses a Dockerrun.aws.json file that
describes how to deploy containers.
The Dockerrun.aws.json file includes three sections:
• AWSEBDockerrunVersion: Set to "2" for multicontainer
Docker environments.
• containerDefinitions: An array of container definitions.
• volumes: Creates mount points in the container instance
that a container can use.

64. Convox

65. Convox
# Initialize your app and create default manifest
> convox init
# Locally build and run your app as declared in the manifest
> convox start
# Create app
> convox apps create my_app
# Deploy app, output ELB DNS name
> convox deploy
[...]
web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

66. Remind Empire
Control layer on top of Amazon ECS that provides a
Heroku like workflow
Any tagged Docker image can be deployed to Empire as
an app
• When you deploy a Docker image to Empire, it will
extract a Procfile from the WORKDIR
• Each process type in the Procfile maps directly to an
ECS Service

67. Remind Empire
Routing layer backed by internal ELBs
• An application that specifies a web process will get an
internal ELB attached to its ECS Service
• When a new internal ELB is created, an associated
CNAME record is created in Route53 under the internal
TLD, enabling service discovery via DNS

68. Using the CLI

69. Using the CLI
Configuring the ECS CLI
Cluster Setup with the ECS CLI
Deploy Compose App with ECS CLI
Scaling with ECS CLI

70. Configuring the ECS CLI
Easily create Amazon ECS clusters & supporting
resources such as EC2 instances
Run Docker Compose configuration files on Amazon
ECS
Available today – http://amzn.to/1jBf45a

71. Configuring the ECS CLI
# Configure the CLI using environment variables
> export AWS_ACCESS_KEY_ID=<my_access_key>
> export AWS_SECRET_ACCESS_KEY=<my_secret_key>
> ecs-cli configure --region us-east-1 --access-key
$AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --cluster
ecs-cli-demo
# Configure the CLI using an existing AWS CLI profile
> ecs-cli configure --region us-west-2 --profile ecs-profile --
cluster ecs-cli-demo

72. Cluster Setup with the ECS CLI
# Creates a new ECS cluster with two container instances in an
existing VPC
> ecs-cli up --capability-iam --keypair my_ecs_keypair --size 2 -
-security-group sg-a12bc34d --vpc vpc-0e9dc8b7 --subnets subnet-
12ab34cd,subnet-56ef78ab --instance-type t2.medium
# Creates a new ECS cluster with one container instance in a new
VPC
> ecs-cli up --capability-iam --keypair my_ecs_keypair --azs us-
east-1a,us-east-1c --cidr 192.169.0.0/24 --port 22 --instance-
type t2.medium

73. Deploy Compose App with ECS CLI
Docker Compose lets you define and run multi-container
applications:
1. Define app environment with Dockerfile
2. Define services that make up your app in docker-
compose.yml
3. Run docker-compose up to start and run entire app

74. Deploy Compose App with ECS CLI
proxy:
build: ./proxy
ports:
- "80:80"
links:
- web
web:
build: ./web
command: bundle exec rails server -b 0.0.0.0
environment:
- SECRET_KEY_BASE=secretkey
expose:
- "3000"

75. Deploy Compose App with ECS CLI
> ecs-cli compose up
> ecs-cli compose ps
> ecs-cli compose service create
> ecs-cli compose service start

76. Scaling with ECS CLI
> ecs-cli scale n
> ecs-cli compose scale n
> ecs-cli compose service scale n

77. Thank you!