Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Introduction to Amazon EC2 Container Service and setting up build pipeline with ECS and Jenkins
1. Introduction to Amazon EC2 Container
Service and setting up build pipeline with
ECS and Jenkins
Swapnil Dahiphale
DevOps Engineer, WhiteHedge Technologies
4. What are containers?
OS virtualization
Process isolation
Automation
Images
Server
Guest OS
Bins/Libs Bins/Libs
App2App1
5. Why use Docker?
With Docker we can solve many problems
• “it works on my machine”
• reducing build & deploy time
• Infrastructure configuration –automation!
• Libs dependency hell
• Cost control and granularity
20. What is ECS?
Easily Manage Clusters for Any Scale
• Nothing to run
• Complete state
• Control and monitoring
• Scale
ECS List* and Describe* APIactions
21. What is ECS?
Flexible Container Placement
• Applications
• Batch jobs
• Multiple schedulers
22. What is ECS?
Designed for use with other AWS services
• Virtual Private Cloud
• Elastic Load Balancing
• Elastic Block Store
• IAM
• CloudTrail
34. Typical user workflow
Run Instances Amazon
EC2
Use custom AMI with
Docker support and
ECS Agent. Instances
will register with
defaultcluster.
35. Typical user workflow
Create TaskDefinition
Declare resource
requirements for
containers
Shared DataVolume
Node.jsApp
Time ofday
App
36. Typical user workflow
Create Service
Declare resource
requirements for
service
Shared DataVolume
Node.jsApp
Time ofday
App
Elastic
Load
Balancing
X 3
Similar to hardware virtualization like ec2 however instead of partitioning machine they isolate processes that are running on single OS.
Useful when you wanna use os kernel to create multiple isolated userspace processes that have constraints on them like CPU and memory.
Docker enables easy use of it, you can create images of containers, that enables automation.
So we are able to define our app, build,share and deploy the image
Put an app code in a container with env required and ship anywhrere, run anywhere,
Its lightweight!
it has the evironment with app, no need to worry about server config, dependencies
Cost in terms of deployment time, efforts, enables automation
Ops guy creates a certified base image, starts with OS required and some patches and utilities required
Base images used to create utility containers
Developers use those containers on their laptops and build a source code,
They do this by creating a manifest file i.e. Dockerfile, the can be versioned and shared.
Custumize image at runtime using environment variables,
Eg you can use same image of database container on dev and test & change database endpoints
that points out
So docker cli is pretty great if you run on a developer laptop
But its challenging to scale that on thousands of containers.
So now u are managing cluster, and managing cluster is pretty hard.
So you need to know the place to put your container, it shouuld have required resources, and it should maintain its state.
How do we manage them
So we need to have orchestration
We saw the pipeline and its hard to do it manually!
Spreading a service across nodes
Making the most use out of a node
Inter-host communication
Different tools available
Biggest challemges in container technologies:
Scaling cluster mgmt system, as no of containers and instances increases how to manage a fleet of them
Because we have to manage availability of cluster, manage how cluster scale without compermizing performance
As we scale app and our fleets, we have to deal with container sprawl because there are so many containers floating around
As move to prod scheduling becomes a change
As prod, security is most imp
People using aws already like these servies and it would be great if we could have integration of cluster mgmt system with them.
Use api’s to run apps or build a platform to run our apps on it
Lets see what ecs with its advatages,
Key featers:
1.Running a reliable distributed app requires 2 ingredients reliable state mgmt and flexible scheduling
ECS handles it, no need to install your cluster mgmt infa.
Through api can get running state quickly.
Scheduler: understands need of app or need to start container with 200 mb of memory on port 80!
Talks with other resources like ELB
Two schedulers- create service and run tasks.
ECS provides cluster state so you can use your own or third party scheduler
Can create a custer in a separate VPC and use scurity groups to isolate them.
Store persistant data using EBS and route traffic using ELB
Cloud trail provides details of every api call for security analysis and tracking
Perfomance is not conpremised while increasing load.
Instances run inside VPC and have security groups
Can use api to integrate with own scedulers or 3rd party.
Can integrate with existing softwares e.g. CI systems
Teminologies:
EC2 instances that are checked into your cluster, normal instance, can ssh
It has a docker daemon
And a ECS agent that is also running as container.
AWS marketplace have AMI with pre-installed above things
Or can create on our own
Cluster is collection of resources, instances and containers running on it
Regional:Can span multiple AZ
Seen as resource pool like cpu n memory
Clusters for env - dev,stage etc
Need not to have container instances of same type
Task: it is what we use for schedulig
Specialy task is grouping of containers
Task can have multiple containers
Defines task:
Which containers to be run,
how much resources it should have
Port mapping
links
I have a docker image with app contained, how do I run