Through a combination of Amazon ECS and open source technologies, customers are able to build portable CI/CD pipelines on AWS. As container based deployments become more complex, they require additional rigging for integration. In this session, we show how popular Apache products like Kakfa, Storm, and Zookeeper are being deployed on top of Amazon ECS. We hear from HERE, a provider of mapping data, technologies, and services to the automotive, consumer, and enterprise sectors about an approach that leverages Consul from Hashicorp and Amazon ECS clusters for short-cycle deployments and tag-based environment promotion.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tom Fuller, Sr. Solutions Architect
Pascal Hahn, HERE Director of Cloud Architecture
Jeremy Brown, HERE Lead Engineer
December 1, 2016
Service Integration Delivery and
Automation Using Amazon ECS
CON308

2. What to Expect from the Session
1. Overview of services being deployed on ECS
and techniques for managing those services
2. An overview of HERE and their Open Location
Platform and Delivery Practices
3. A detailed explanation of the lessons HERE
has learned automating their deployments on
top of ECS

3. Disclaimer
• Best practices are often relative and
time bound, these examples
represent experiences over the last
year
• Docker and the surrounding
ecosystem is moving fast and we are
only scratching the surface
• Innovations are not frozen in time!

4. DevOps - PSA

5. Why Is This Hard?
• Configuration
• Service discovery
• Complexity of cluster types running on
containers
• Volume of clusters
• Scalability
• High availability

6. Apache Foundation Services
Kafka
• Scalable and efficient
streaming ingestion
• Dependency on
Zookeeper
• Tight integration with
Storm
• Scales up with ease
• Works best with locally
attached storage
• Has different modes of
operation (high
throughput vs. high
resiliency)
Storm
• Scalable and efficient
distributed real-time
computation system
• Dependency on
Zookeeper
• You deploy topologies
that are made up of
spouts and bolts
• Spouts are the
abstraction that enables
integration
Zookeeper
• Centralized service for
distributed sync and
configuration
• Draws initial inspiration
from Chubby
• Very sensitive to latency
• Typical configurations
involve 3 or 5 nodes
• Uses Zab for consensus

7. What is Service Discovery?
• The world of microservices is
dynamic
• Containers bring about a layer of
routing
• You need a way to find these
services
• You need to know if the service is
healthy
• How do we find things on the
internet today? Why wouldn’t that
still work?
Service Registry
Service A Service B
publishlookup
bind
Service Health
health checkhealth check
delete

8. Service Discovery on AWS
• A closed DNS solution
(Amazon Route 53) to the
rescue
• Isolation benefits with VPC
and private zones
• How do you maintain the
entries
• Amazon CloudWatch +
AWS Lambda
• ecssd_agent.go
• What about health checks?

9. Listening to Docker Events
endpoint := "unix:///var/run/docker.sock”
startFn := func(event *docker.APIEvents) error {
var err error
container, err := dockerClient.InspectContainer(event.ID)
logErrorAndFail(err)
port, service := getNetworkPortAndServiceName(container, true)
if port != "" && service != "" {
sum = 1
for {
if err = createDNSRecord(service, event.ID, port); err == nil {
break
}
if sum > 8 {
log.Error("Error creating DNS record")
break
}
time.Sleep(time.Duration(sum) * time.Second)
sum += 2
}
}
fmt.Println("Docker " + event.ID + " started")
return nil
}

10. Configuration Data on Amazon ECS
• AWS Identity & Access Management (IAM) roles
for tasks to the rescue
• Limit access to specific configuration sources
with policy
• Requires version 1.11 of the container agent
and a supported SDK/CLI version
• VPC S3 endpoints provide an additional layer of
security
• The rest is basically standard Amazon EC2
goodness
• Temporary rotated credentials
• Strict policy enforcement on Amazon S3
• Encryption at rest with AWS KMS

11. HERE Open Location Platform
Analytics Execution Environment
Pascal Hahn & Jeremy Brown
2016

12. The Open Location Platform will be the epicenter of an ecosystem
that creates value through innovation and collaboration
Automotive
Enterprise IoT
Consumer IoT

13. Location is the glue that will connect disparate data sources.

14. Core components critical to realizing the value of location data
Ingestion
Bring your own data
all while maintaining
control of privacy and
access controls
Enrichment
Combine map
content with historic
and real-time data to
add location context
to your data and
services
Publish
Promote and share
your enriched data
and services with
ecosystem partners
or through your own
channels
Processing
Access off-the-shelf
algorithms while
applying historic,
real-time and
predictive analytics
to your data

15. Analytics Execution Environment
• Analytics data and
processing
• Data buckets
• Processing clusters
• Processing chains
• Users
• Operational services
• Monitoring
• Cost
• Performance
• Availability
• Log management
• Network connectivity (user
VPN)

16. AEE – Architectural overview
AEE Console (Web-UI)
AEE RESTFul API
AEE UI Framework
Service Deployment API
Resource Management
API
Deployment Orchestration
Framework
HERE
AAA
IAM
ECS
EMR
S3 ELB
HERE
Frontend

17. Service Delivery Design

18. Service Delivery Architecture
CI/CD Servers
Image Management
Deploy Strategy
Amazon ECS Amazon ECS

19. Service Discovery and
Registration

20. Overview of Consul and Vault
Consul
• Distributed HA Service Registry
• Service Discovery (DNS)
• KV storage
• Health checks
Vault
• Distributed HA secrets management
• Encrypted K/V storage
• Secrets and Certificate generation and
revocation

21. Service Registration and Discovery at HERE
For HEREs Service Delivery Platform, we organize the running services
into namespaces.
Basenamespace
• "/team/env/app”
Service Name
• tomcat
• mysql
team1-dev-app1-mysql
team1-dev-app1-tomcat
…
team2-prd-app4-mysql
team2-prd-app4-tomcat

22. Namespace Hierarchy
Root /
team1 team2
prddevprddev
app2app1app2app1 app4app3app4app3
mysql tomcatmysql tomcat

23. Namespaces help make policies sensible
IAM Policy

24. Namespaces help make policies cohesive
Consul Policy

25. How Zookeeper/Kafka/Storm discover each other
The trio always discover each other by base-namespace
team1-dev-app1-
zookeeper*
team1-dev-app1-
storm*
team1-dev-app1-
kafka*

26. Container Management

27. Configuring containers and services at run time
When configuring containers at run time, there a couple of
choices for how to do it.
Configurable containers
Containers are configured by the
scheduler at launch usually via envars
(-e flags)
Self-configuring containers
Containers are configured at runtime
by a templating mechanism usually
reading config data from a data-store

28. Configurable Containers
How to implement it?
• Store configuration in a repo or bucket available to the CI/CD server
that triggers ECS task updates.
Pros
• Easy to implement
• Containers are immutable
• Easier to reason about
Cons
• Envars are not secure
• Config Update requires restart
• Your scheduler is also your
configuration manager

29. Self-Configuring Containers
How to implement it?
• Base Containers embed code that retrieves app configuration from distributed
• Data-stores like Consul K/V or etcd and/or a secrets store like Vault or Keywhiz.
• Values and secrets can be written out to config files via templates.
Pros
• Containers and processes can
be updated on the fly
• Nginx use case
• Secrets are stored securely
• Configs can be easily viewed
Cons
• It takes a lot more work up front
• Local development is more
complex
• Requires Consul and Vault on
dev machines

30. Managing container releases
and Hierarchies at scale

31. Register Docker Image for Updates

32. Discover versions and relationships from the Git
Repository
here/centos-7
FROM centos/centos-7:v4
here/java-8
FROM here/centos-7:v3
here/python:v8
FROM here/centos-7:v3
here/flask:v10
FROM here/python:v8
here/tomcat-8:v8
FROM here/java-8:v12
here/tomcat-7:v12
FROM here/java-8:v12

33. Discover versions and relationships from the
Docker Registry
here/centos-7:v3
here/java-8:v12 here/python:v8
here/flask:v10here/tomcat-8:v8
here/tomcat-7:v12
here/centos-7:v3
here/centos-7:v3
here/java-8:v12
here/java-8:v12
here/java-8:v12
here/python:v8
here/python:v8
here/python:v8
here/tomcat-7:v12
here/tomcat-7:v12
here/tomcat-7:v12
here/tomcat-7:v12
here/tomcat-7:v12
here/tomcat-8:v8
here/tomcat-8:v8
here/tomcat-8:v8
here/tomcat-8:v8
here/tomcat-8:v8
here/tomcat-8:v8
here/flask:v10
here/flask:v10
here/flask:v10
here/flask:v10
here/flask:v10
here/flask:v10

34. Update to the latest Parent
New Image
here/centos-7:v4
java-8 Dockerfile
- FROM here/centos-7:v4
+FROM here/centos-7:v5
Python Dockerfile
- FROM here/centos-7:v4
+FROM here/centos-7:v5
tomcat-7 Dockerfile
FROM here/java-8:v12
tomcat-8 Dockerfile
FROM here/java-8:v12
flask Dockerfile
FROM here/python:v8

35. Update to the latest Parent
New Image
here/centos-7:v4
java-8 Dockerfile
FROM here/centos-7:v5
Python Dockerfile
FROM here/centos-7:v5
tomcat-7 Dockerfile
- FROM here/java-8:v12
+FROM here/java-8:v13
tomcat-8 Dockerfile
- FROM here/java-8:v12
+FROM here/java-8:v13
flask Dockerfile
- FROM here/python:v8
+FROM here/python:v9

36. How a Docker container get versioned
Each master verify build results in a Docker image that is tagged as
testing.
testing-$symanticVersion-$timestamp-$gitrev
testing-1.6.5-1477661204-d871ef9
This results in a clean natural order and all tags can be traced back to
their origins in code or build.
Once a Docker image passes all acceptance tests, it gets retagged
stable.
stable-1.6.5-1477661204-d871ef9

37. Managing Service releases at
scale

38. How Docker containers get promoted
Collections of images that pass acceptance testing together get released
by pushing a bill of materials to an artifact repository.
ZOOKEEPER=here/zookeeper:stable-1.6.5-1477661204-d871ef9
KAFKA=here/kafka:stable-0.10.1.0-1476105328-79c3c0c
STORM=here/storm:stable-1.0.2-1476193847-7fe4563
Now this BOM can be used by any other environment based on its own
specific configs stored in Consul and Vault.

39. Releasing Services through CI/CD pipelines
HERE separates CI into distinct stages; all development happens on the
master.
Pre-submit
verify
Master
verify
Product
acceptance
test
System
acceptance
test
Customer
integration
test
Production

40. Some statistics on speed of deployment
• ECS has taken one of the slowest parts of our CI/CD pipeline and
made it one of the fastest.
ECS
12 to 15 minutes
95% success rate
30 to 60 seconds
99%+ success rate

41. Thank you!

42. HERE makes it quick and easy to enable location based
solutions
Visit Booth 2316
Geocoding – New on the
AWS Marketplace
With HERE Geocoder, plan to
to add geocoding services to
to your app. Each plan allows
allows you to use its features
features for a single app or
website. To use HERE
platform for more than one
app or website, simply
register for an additional plan
plan for each additional app
app or website.
REST APIs
Add geocoding, positioning,
routing and more to any
connected device. The HERE
Location Platform brings
advanced features to low-power
hardware through a set of flexible
services accessible via REST APIs.
Positioning via REST APIs
Find the location of your apps and hardware
around the work, indoors and outdoors.
HERE has network positioning with global
cell and Wi-Fi coverage, including China,
for low-cost location use-cases, and
advanced assistance features to increase
accuracy and decreased time-to-first-fix.
High accuracy network positioning allows
you to detect location even if your hardware
doesn’t support GNSS.
Up for a Challenge?
HERE Top 50 Coding Challenge
Let us prove how the HERE cloud makes developing location-based services
fast and easy. The first 50 developers to participate in this contest receive a
premium miniature drone giveaway at the HERE booth during Amazon
Re:invent. Stop by our booth to find out more!

43. Remember to complete
your evaluations!