What do companies with internal platforms have to change to succeed in the cloud? The five pillars at the heart of IT solutions in the cloud are automation, fault tolerance, horizontal scalability, security, and cost-effectiveness. This talk discusses tools that facilitate the development and automate the deployment of secure, highly available microservices. The tools were developed using AWS CloudFormation, AWS SDKs, AWS CLI, Amazon RDS, and various open-source software such as Docker. The talk provides concrete examples of how these tools can help developers and architects move from beginning/intermediate AWS practitioners to cloud deployment experts.
9. Theory of Cloud
Automated
Elastic
Highly Available
Security
Software defined everything
Unlimited scale + pay-as-you-go
Horizontally Scalable
Multi-AZ/region + shards/replicas
Provision more like things any time
“Do over” + Correct by construction
10. Theory of Cloud Cloud Architecture
Automated Higher-Order Automation
Elastic Ephemeral Environments
Highly Available Fault Tolerant
Security Secure by Construction
Horizontally Scalable Parallel, Commodity
⇒
12. Fallacies of Internal Apps
1. The hardware is reliable
2. The network is reliable
3. The database is reliable
4. Other services are available
5. Inside the network is secure
6. …
Fault Tolerant
13. Fault Tolerant
Fallacies of 1st Generation Cloud
1. Other people’s fault tolerant
code is actually fault tolerant
2. Everything is stateless
3. Everything can be retried
4. Applications should handle all
faults
5. Data is magically handled by
someone else
15. A Do-Over for Secure by Construction
Secure by Assumption
Secure by Design
Security Automation
16. Horizontally Scalable
1. The overhead of scaling
grows at most linearly with
additional nodes
2. Reads and writes both
scale out
3. The system can continue to
provide this scalability
under loss of any node
* This (CAP) requires apps to
understand conflicts
30. Stax
$ ./stax --help
Usage: stax [OPTIONS] COMMAND [COMMAND_ARGS]
add Add functionality to an existing VPC
auto-services Lanch multiple services on fleet using template/NAME.services file
check Run various tests against an existing stax
clean Remove keys and buckets of non-existant stacks
connect [TARGET] Connect to bastion|gateway|service in the VPC stax over SSH
create Create a new VPC stax in AWS
describe Describe the stax created from this host
delete Delete the existing VPC stax
dockerip-update Fetch docker IP addresses and update related files
fleet Run various fleetctl commands against the fleet cluster
help Output this message
history View history of recently created/deleted stax
list List all completely built and running stax
rds PASSWORD Create an RDS instance in the DB subnet
rds-delete RDSIN Delete RDS instance RDSIN
remove ADD Remove the previously added ADD
services List servers that are available to run across a stax
slack Post usage report to Slack, define hook in stax.config
sleep Turn on/off bastion host which allows ssh access into the VPC
start SERVICE Start service SERVICE in the fleet cluster
test Automated test to exercise functionality of stax
update Update an existing VPC with changes from Cloudformation
validate Validate CloudFormation template
For more help, check the docs: https://github.com/MonsantoCo/stax
Create and
manage
CloudFormation
stacks in AWS
40. A modern language for software engineering
Abstract Data Types (ADTs)
Enforced Immutability
Pattern Matching & Destructuring
Assignment
Type-Level Programming
Futures, Actors, Async
Type classes
Scala, Haskell, Swift, OCaML, SML
Scala, Haskell, Clojure, Erlang, OCaML,
SML
CoffeeScript, Scala, Haskell, Swift, OCaML,
Erlang, SML
Haskell, Scala, C++
Erlang, Scala, Java
Haskell, Scala, ~OCaML
Hybrid OO/FP
Provides transition from and backward compatibility with Java
41. Advanced Abstractions
Algebraic Data Types (ADTs)
Enforced Immutability
Pattern Matching & Destructuring
Assignment
Type-Level Programming
Futures, Actors, Async
Type classes
Scala: A Modern Language for Software Engineering
Advanced Type Constraints
Advanced Generics & Variance
Higher Kinds
F-bounded Polymorphism
Self-Types
Type Projections
Type Members
Path Dependent Types
Type Refinements
Turing-complete!
45. Project-as-a-Service 2 – Simple Service Template
Runs giter8 to create a fully functional service written in
Scala based off our current best practices:
• Standard libraries (Slick, Spray, Akka, etc.) for
microservices
• Automated tests with ScalaTest
• Administrative REST endpoints
• Built in (remote) logging and metric capabilities
• Auto-Docker-ization
• Local Vagrant environment
48. fleet
Router
Route Updater
Registrator
A commit is made to GitHub1
1
https://github.com/MonsantoCo/etcd-aws-cluster
https://github.com/MonsantoCo/docker-aws
https://github.com/MonsantoCo/fleet-client
54. fleet
Router
Route Updater
Registrator
When a request is received, the router determines the current revision for the service as
well as the location of the service.
7
7
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service v1 rev1
10.183.0.100:8080
55. fleet
Router
Route Updater
Registrator
Next commit (rev 2) is received, Jenkins will test/build/push and look up the revision from
etcd. The revision is newer so it continues but does not update the current revision.
8
8
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service v1 rev1
service-1:2
10.183.0.100:8080
56. fleet
Router
Route Updater
Registrator
Jenkins deploys the new container to fleet. It runs side-by-side with the previous
revision at a different location.
9
9
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service v1 rev1
service-1:2
service v1 rev2
10.183.0.100:8081
10.183.0.100:8080
57. fleet
Router
Route Updater
Registrator
Registrator notices the new service is deployed and registers the location in etcd under
a different key.
10
10
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2
10.183.0.100:8081
10.183.0.100:8080
58. fleet
Router
Route Updater
Registrator
Traffic continues to flow to the old service as the current revision has not changed.11
11
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2
10.183.0.100:8081
10.183.0.100:8080
59. fleet
Router
Route Updater
Registrator
Traffic can be directed to a particular version by using a header for testing purposes.12
12
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2
X-Service-Revision: 2
10.183.0.100:8081
10.183.0.100:8080
60. fleet
Router
Route Updater
Registrator
Periodically, Route Updater queries etcd to look for cases where there is a revision
deployed that is newer than the current route.
13
service-1:1
service-1 => 1
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2 13
10.183.0.100:8081
10.183.0.100:8080
61. fleet
Router
Route Updater
Registrator
If there is a newer revision, route updater will attempt to call the smoketest endpoint. If
this returns true, it updates the current route.
14
service-1:1
service-1 => 2
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2 14
/admin/smoketest
10.183.0.100:8081
10.183.0.100:8080
62. fleet
Router
Route Updater
Registrator
Now traffic will start flowing to the new revision of the service automatically.15
service-1:1
service-1 => 2
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2
15
10.183.0.100:8081
10.183.0.100:8080
63. fleet
Router
Route Updater
Registrator
Route Updater will notice that there is a stale revision running. It will instruct the service
to cleanly exit by making a call to the /admin/shutdown endpoint.
16
service-1:1
service-1 => 2
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]service v1 rev1
service-1:2
service v1 rev2
16
/admin/shutdown
10.183.0.100:8081
10.183.0.100:8080
64. fleet
Router
Route Updater
Registrator
Registrator will notice the container is no longer running and remove its location from
etcd.
17
service-1:1
service-1 => 2
service-1-1 =>
[10.183.0.100:8080]
service-1-2 =>
[10.183.0.100:8081]
service-1:2
service v1 rev2
17
10.183.0.100:8081
65. fleet
Router
Route Updater
Registrator
The system continues as-is until a new revision is deployed.18
service-1:1
service-1 => 2
service-1-2 =>
[10.183.0.100:8081]
service-1:2
service v1 rev2
10.183.0.100:8081
66. Comprehensive
Service – log4j
Container – logspout
CoreOS – journal forwarder
Bastion/NAT – rsyslog
ELB – S3 (ELK coming soon)
S3 – S3 (ELK coming soon)
CloudTrail – S3 → TrailDash
RDS – (coming soon)
Logging with ScalaLogging and ELK
Easy to use
• Standard ScalaLogging interface
• Auto custom formats (stack traces)
• JSON-format log messages
• Direct-to-ELK writing
• Standard Fields (container ID, code
version, service name, etc)
67. Instrumentation & Shipping
• Kamon to Prometheus
Exporter, preserves more
metrics than Prometheus JVM
• Improved tracing
• Improved complex data
mapping
• Periodically collect and push
Spray metrics to Kamon
Automating Kamon and Prometheus
Auto-discovery, Dashboards, Alerts
• Custom Docker containers with
more automation – etcd
discovery
• Custom default dashboards
• Auto EC2/EBS/RDS standup
• OAuth integration
• SNS notification integration
• Default Alerts
https://github.com/MonsantoCo/spray-kamon-metrics
69. Improvements & Evolution
AWS Service Catalog – API?
EC2 Container Service
AWS IAM
• EC2 CS Roles
• RDS Roles – per VPC/DB Subnet Groups
Amazon API Gateway
VPC Flow Logs – CloudFormation support?
Inverting control for deployment
CloudFormation update predictability
IAM role
Amazon RDS
Amazon EC2
Container
Service
72. Acknowledgements
Larry Anderson
Chris Coffman
TJ Corrigan
Phil Cryer
Dave D’Alessandro
Daniel Solano Gómez
Justin Honold
Kyle Jones
Jessica Kerr
Kevin Meredith
Jorge Montero
Brian Rodgers
Chris Shafer
Niranjan Vengavasi
Dick Wall
Russ Wilson
Stuart Wong
75. Related Sessions
ARC309 - From Monolithic to Microservices: Evolving
Architecture Patterns in the Cloud
Thursday, Oct 8, 4:15 PM - 5:15 PM – Palazzo N
MBL203 - From Drones to Cars: Connecting the
Devices in Motion to the Cloud
Friday, Oct 9, 10:15 AM - 11:15 AM – Delfino 4005