SlideShare a Scribd company logo
1 of 19
Download to read offline
ElasticSearch on AWS
Continuously Deployed,
Immutable and Stateful
About me
● Philipp Garbe (@pgarbe)
● Husband
● Father
● AutoScout24
● Former Microsoft-Fanboy
● Now Docker-Fanboy ;)
● Use case: Logging for AS24
● Challenge: How to handle the state?
○ Immutable
○ Continuously Deployed
○ Stateful
● Next challenges
What to expect
Project
Tatsu
Unified
Logs
CC
STRATEGIC
GOALS
Goals of the business side
ARCHITECTURAL
PRINCIPLES
High-Level Principles
DESIGN AND DELIVERY
PRINCIPLES
Tactical measures
REDUCE TIME TO MARKET
Speed, Fast Feedback
COST EFFICIENCY
Collect metrics to allow decisions cost vs. value.
SUPPORT DATA-DRIVEN DECISIONS
Listen to users and validate hypothesis.
Provide as many relevant metrics & data as possible.
YOU BUILT IT, YOU RUN IT
The team is responsible for shaping, building, running and
maintaining its products. Fast feedback from live and
customers helps us to continuously improve.
ORGANIZED AROUND BUSINESS CAPABILITIES
Build teams around products not projects. Follow the
domain and respect bounded contexts. Inverse Conway
Maneuver.
LOOSELY COUPLED
By default avoid sharing and tight coupling, except for the
big things in common. Don’t create the next monolith.
MACRO AND MICRO ARCHITECTURE
Clear separation. Autonomous micro services within the
rules and constraints of the macro architecture.
AWS FIRST
Favor AWS platform service over managed service, over
self-hosted OSS, over self-rolled solutions.
DATA-DRIVEN / METRIC-DRIVEN
Collect metrics from processes and applications. Analyze,
alert and act on them.
ELIMINATE ACCIDENTAL COMPLEXITY
Strive to keep it simple. Focus on essential complexity.
You build one, you delete one.
AUTONOMOUS TEAMS
Make fast local decisions. Be responsible. Know your
boundaries. Share findings.
INFRASTRUCTURE AS CODE
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.
COLLABORATION CULTURE
Engineers from all backgrounds work together in
collaborative teams as engineers and share
responsibilities. No silos.
BE BOLD
Go into production early. Value monitoring over tests.
Recover and learn. Optimize for MTTR not MTBF.
SECURITY, COMPLIANCE AND DATA PRIVACY
Security must be included from the beginning and
everybody’s concern. Keep data-privacy in mind.
CONTAINMENT AND BOUNDARIES
Align blast radius and vendor lock-in with the boundaries
of the organization or business capabilities.
Version 1.0
Icons made by Freepik from www.flaticon.com are licensed under CC BY 3.0
https://github.com/autoscout24/scout24-it-principles
INFRASTRUCTURE AS CODE
Automate everything: Reproducible, traceable and tested.
Immutable servers over snowflake servers.
Some numbers
12 nodes (m4.4xlarge, EBS gp2 á 2000 GB)
9 indices (one per day)
648 shards
7 Billion docs (new documents peak: 2,500 docs / sec)
16.99 TB of data
Challenge: How to handle the state?
Immutable Servers
Rake
CloudFormation
ElasticSearch configuration
Continuously Deployed
● One CloudFormation stack per node
● Rolling updates through rake
● Every commit goes to production
Stateful
Keep the data!
Monitoring
● CloudWatch
○ Cron-Lambda sends /_cluster/health metrics to CloudWatch
○ CloudWatch alarms based on these metrics
● DataDog
○ Deep Dive
Next challenges
No AvailabilityZone awareness
Next challenges
Mapping issues
Next challenges
Long-running deployments (2-8h)
?Questions
Philipp Garbe
http://garbe.io
@pgarbe
https://github.com/pgarbe

More Related Content

What's hot

AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
Amazon Web Services
 

What's hot (20)

(SEC309) Amazon VPC Configuration: When Least Privilege Meets the Penetration...
(SEC309) Amazon VPC Configuration: When Least Privilege Meets the Penetration...(SEC309) Amazon VPC Configuration: When Least Privilege Meets the Penetration...
(SEC309) Amazon VPC Configuration: When Least Privilege Meets the Penetration...
 
(SEC326) Security Science Using Big Data
(SEC326) Security Science Using Big Data(SEC326) Security Science Using Big Data
(SEC326) Security Science Using Big Data
 
Security Day IAM Recommended Practices
Security Day IAM Recommended PracticesSecurity Day IAM Recommended Practices
Security Day IAM Recommended Practices
 
Mastering azure devOps - Dot Net Tricks
Mastering azure devOps - Dot Net TricksMastering azure devOps - Dot Net Tricks
Mastering azure devOps - Dot Net Tricks
 
(SEC313) Security & Compliance at the Petabyte Scale
(SEC313) Security & Compliance at the Petabyte Scale(SEC313) Security & Compliance at the Petabyte Scale
(SEC313) Security & Compliance at the Petabyte Scale
 
(SEC325) Satisfy PCI Obligations While Continuing to Innovate
(SEC325) Satisfy PCI Obligations While Continuing to Innovate(SEC325) Satisfy PCI Obligations While Continuing to Innovate
(SEC325) Satisfy PCI Obligations While Continuing to Innovate
 
Security and Compliance
Security and ComplianceSecurity and Compliance
Security and Compliance
 
(SEC312) Reliable Design & Deployment of Security & Compliance
(SEC312) Reliable Design & Deployment of Security & Compliance(SEC312) Reliable Design & Deployment of Security & Compliance
(SEC312) Reliable Design & Deployment of Security & Compliance
 
Google App Engine - Simple Introduction
Google App Engine - Simple IntroductionGoogle App Engine - Simple Introduction
Google App Engine - Simple Introduction
 
Kibana Tutorial | Kibana Dashboard Tutorial | Kibana Elasticsearch | ELK Stac...
Kibana Tutorial | Kibana Dashboard Tutorial | Kibana Elasticsearch | ELK Stac...Kibana Tutorial | Kibana Dashboard Tutorial | Kibana Elasticsearch | ELK Stac...
Kibana Tutorial | Kibana Dashboard Tutorial | Kibana Elasticsearch | ELK Stac...
 
(New)SQL on AWS: Aurora serverless
(New)SQL on AWS: Aurora serverless(New)SQL on AWS: Aurora serverless
(New)SQL on AWS: Aurora serverless
 
(SEC301) Encryption and Key Management in AWS | AWS re:Invent 2014
(SEC301) Encryption and Key Management in AWS | AWS re:Invent 2014(SEC301) Encryption and Key Management in AWS | AWS re:Invent 2014
(SEC301) Encryption and Key Management in AWS | AWS re:Invent 2014
 
AWS Cloud trail
AWS Cloud trailAWS Cloud trail
AWS Cloud trail
 
(DEV306) Building Cross-Platform Applications Using the AWS SDK for JavaScrip...
(DEV306) Building Cross-Platform Applications Using the AWS SDK for JavaScrip...(DEV306) Building Cross-Platform Applications Using the AWS SDK for JavaScrip...
(DEV306) Building Cross-Platform Applications Using the AWS SDK for JavaScrip...
 
(SEC315) NEW LAUNCH: Get Deep Visibility into Resource Configurations | AWS r...
(SEC315) NEW LAUNCH: Get Deep Visibility into Resource Configurations | AWS r...(SEC315) NEW LAUNCH: Get Deep Visibility into Resource Configurations | AWS r...
(SEC315) NEW LAUNCH: Get Deep Visibility into Resource Configurations | AWS r...
 
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
AWS March 2016 Webinar Series - Best Practices for Managing Security Operatio...
 
Getting Started with AWS IoT
Getting Started with AWS IoTGetting Started with AWS IoT
Getting Started with AWS IoT
 
Aws Overview
Aws OverviewAws Overview
Aws Overview
 
Hybrid computing Azure with Azure Stack by Atcetera // Azure Multi-Cloud
Hybrid computing Azure with Azure Stack by Atcetera // Azure Multi-CloudHybrid computing Azure with Azure Stack by Atcetera // Azure Multi-Cloud
Hybrid computing Azure with Azure Stack by Atcetera // Azure Multi-Cloud
 
Durable Azure Functions
Durable Azure FunctionsDurable Azure Functions
Durable Azure Functions
 

Viewers also liked

Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...
Spark Summit
 
The modern analytics architecture
The modern analytics architectureThe modern analytics architecture
The modern analytics architecture
Joseph D'Antoni
 
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
DB Tsai
 

Viewers also liked (20)

Large scale near real-time log indexing with Flume and SolrCloud
Large scale near real-time log indexing with Flume and SolrCloudLarge scale near real-time log indexing with Flume and SolrCloud
Large scale near real-time log indexing with Flume and SolrCloud
 
Building an ETL pipeline for Elasticsearch using Spark
Building an ETL pipeline for Elasticsearch using SparkBuilding an ETL pipeline for Elasticsearch using Spark
Building an ETL pipeline for Elasticsearch using Spark
 
Real time analytics using Hadoop and Elasticsearch
Real time analytics using Hadoop and ElasticsearchReal time analytics using Hadoop and Elasticsearch
Real time analytics using Hadoop and Elasticsearch
 
Spark Summit EU talk by Tug Grall
Spark Summit EU talk by Tug GrallSpark Summit EU talk by Tug Grall
Spark Summit EU talk by Tug Grall
 
Apache Flume
Apache FlumeApache Flume
Apache Flume
 
2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator2017 02-07 - elastic & spark. building a search geo locator
2017 02-07 - elastic & spark. building a search geo locator
 
Apache Flume - DataDayTexas
Apache Flume - DataDayTexasApache Flume - DataDayTexas
Apache Flume - DataDayTexas
 
Introducing Akka
Introducing AkkaIntroducing Akka
Introducing Akka
 
Combine Apache Hadoop and Elasticsearch to Get the Most of Your Big Data
Combine Apache Hadoop and Elasticsearch to Get the Most of Your Big DataCombine Apache Hadoop and Elasticsearch to Get the Most of Your Big Data
Combine Apache Hadoop and Elasticsearch to Get the Most of Your Big Data
 
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlibElasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlib
 
Feb 2013 HUG: Large Scale Data Ingest Using Apache Flume
Feb 2013 HUG: Large Scale Data Ingest Using Apache FlumeFeb 2013 HUG: Large Scale Data Ingest Using Apache Flume
Feb 2013 HUG: Large Scale Data Ingest Using Apache Flume
 
Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...
Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...
Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...
 
Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
 
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
Building a Dataset Search Engine with Spark and Elasticsearch: Spark Summit E...
 
Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...Realtime Analytical Query Processing and Predictive Model Building on High Di...
Realtime Analytical Query Processing and Predictive Model Building on High Di...
 
Integrating Elastic and Apache Spark - Elastic London Meetup (2015-09-24)
Integrating Elastic and Apache Spark - Elastic London Meetup (2015-09-24)Integrating Elastic and Apache Spark - Elastic London Meetup (2015-09-24)
Integrating Elastic and Apache Spark - Elastic London Meetup (2015-09-24)
 
963
963963
963
 
The modern analytics architecture
The modern analytics architectureThe modern analytics architecture
The modern analytics architecture
 
Hadoop and Spark Analytics over Better Storage
Hadoop and Spark Analytics over Better StorageHadoop and Spark Analytics over Better Storage
Hadoop and Spark Analytics over Better Storage
 
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
 

Similar to ElasticSearch on AWS

Similar to ElasticSearch on AWS (20)

Building Microservices in the cloud - GOTO Nights Berlin 2016
Building Microservices in the cloud - GOTO Nights Berlin 2016Building Microservices in the cloud - GOTO Nights Berlin 2016
Building Microservices in the cloud - GOTO Nights Berlin 2016
 
Building Microservices in the cloud - Software Architecture Summit 2016
Building Microservices in the cloud - Software Architecture Summit 2016Building Microservices in the cloud - Software Architecture Summit 2016
Building Microservices in the cloud - Software Architecture Summit 2016
 
Microservices in the cloud at AutoScout24
Microservices in the cloud at AutoScout24Microservices in the cloud at AutoScout24
Microservices in the cloud at AutoScout24
 
Highway to heaven - Microservices Meetup Dublin
Highway to heaven - Microservices Meetup DublinHighway to heaven - Microservices Meetup Dublin
Highway to heaven - Microservices Meetup Dublin
 
GOTO Berlin 2016
GOTO Berlin 2016GOTO Berlin 2016
GOTO Berlin 2016
 
Microservices in der Cloud - Software Architecture Summit Berlin 2016
Microservices in der Cloud - Software Architecture Summit Berlin 2016Microservices in der Cloud - Software Architecture Summit Berlin 2016
Microservices in der Cloud - Software Architecture Summit Berlin 2016
 
AWS re:Invent 2016: [JK REPEAT] The Enterprise Fast Lane - What Your Competit...
AWS re:Invent 2016: [JK REPEAT] The Enterprise Fast Lane - What Your Competit...AWS re:Invent 2016: [JK REPEAT] The Enterprise Fast Lane - What Your Competit...
AWS re:Invent 2016: [JK REPEAT] The Enterprise Fast Lane - What Your Competit...
 
Reactive Microservices Roadshow Berlin
Reactive Microservices Roadshow BerlinReactive Microservices Roadshow Berlin
Reactive Microservices Roadshow Berlin
 
Building Microservices in the cloud at AutoScout24
Building Microservices in the cloud at AutoScout24Building Microservices in the cloud at AutoScout24
Building Microservices in the cloud at AutoScout24
 
Transforming to OpenStack: a sample roadmap to DevOps
Transforming to OpenStack: a sample roadmap to DevOpsTransforming to OpenStack: a sample roadmap to DevOps
Transforming to OpenStack: a sample roadmap to DevOps
 
Serverless is a win for businesses, not just developers
Serverless is a win for businesses, not just developersServerless is a win for businesses, not just developers
Serverless is a win for businesses, not just developers
 
When to Code / Config / Config + Code in Salesforce - Nikunj Doshi
When to Code / Config / Config + Code in Salesforce - Nikunj DoshiWhen to Code / Config / Config + Code in Salesforce - Nikunj Doshi
When to Code / Config / Config + Code in Salesforce - Nikunj Doshi
 
Highway to heaven - Voxxed Days Belgrade
Highway to heaven - Voxxed Days BelgradeHighway to heaven - Voxxed Days Belgrade
Highway to heaven - Voxxed Days Belgrade
 
Data science tools of the trade
Data science tools of the tradeData science tools of the trade
Data science tools of the trade
 
DevOpsDays Jakarta Igites
DevOpsDays Jakarta IgitesDevOpsDays Jakarta Igites
DevOpsDays Jakarta Igites
 
AWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API Calls
AWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API CallsAWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API Calls
AWS Partner: Grindr: Aggregate, Analyze, and Act on 900M Daily API Calls
 
July 29: Achieve Stakeholder Alignment on Governance: Navigator by Pivotal Labs
July 29: Achieve Stakeholder Alignment on Governance: Navigator by Pivotal LabsJuly 29: Achieve Stakeholder Alignment on Governance: Navigator by Pivotal Labs
July 29: Achieve Stakeholder Alignment on Governance: Navigator by Pivotal Labs
 
Approaching risk management with your head in the cloud
Approaching risk management with your head in the cloudApproaching risk management with your head in the cloud
Approaching risk management with your head in the cloud
 
Optimize your CI/CD with GitLab and AWS
Optimize your CI/CD with GitLab and AWSOptimize your CI/CD with GitLab and AWS
Optimize your CI/CD with GitLab and AWS
 
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
(SPOT205) 5 Lessons for Managing Massive IT Transformation Projects
 

More from Philipp Garbe

More from Philipp Garbe (9)

Run Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS MeetupRun Jenkins as Managed Product on ECS - AWS Meetup
Run Jenkins as Managed Product on ECS - AWS Meetup
 
Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?Is Platform Engineering the new Ops?
Is Platform Engineering the new Ops?
 
Managed Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECSManaged Container Orchestration with Amazon ECS
Managed Container Orchestration with Amazon ECS
 
Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017Deliver Docker Containers Continuously on AWS - QCon 2017
Deliver Docker Containers Continuously on AWS - QCon 2017
 
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
Deliver Docker Containers Continuously On AWS - DevOpsCon Munich 2016
 
Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24
Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24
Finding Cars and Hunting Down Logs - ElasticSearch @AutoScout24
 
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016Docker Container automatisiert nach AWS deployen  - Continuous Lifecycle 2016
Docker Container automatisiert nach AWS deployen - Continuous Lifecycle 2016
 
Deliver docker containers continuously on aws
Deliver docker containers continuously on awsDeliver docker containers continuously on aws
Deliver docker containers continuously on aws
 
DockerCon 2016 Seattle Recap
DockerCon 2016 Seattle RecapDockerCon 2016 Seattle Recap
DockerCon 2016 Seattle Recap
 

Recently uploaded

Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort ServiceCall Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
Sheetaleventcompany
 
AWS Community DAY Albertini-Ellan Cloud Security (1).pptx
AWS Community DAY Albertini-Ellan Cloud Security (1).pptxAWS Community DAY Albertini-Ellan Cloud Security (1).pptx
AWS Community DAY Albertini-Ellan Cloud Security (1).pptx
ellan12
 
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
soniya singh
 
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
Chandigarh Call girls 9053900678 Call girls in Chandigarh
 
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
@Chandigarh #call #Girls 9053900678 @Call #Girls in @Punjab 9053900678
 
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
soniya singh
 
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRLLucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
imonikaupta
 

Recently uploaded (20)

Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort ServiceCall Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
Call Girls in Prashant Vihar, Delhi 💯 Call Us 🔝9953056974 🔝 Escort Service
 
On Starlink, presented by Geoff Huston at NZNOG 2024
On Starlink, presented by Geoff Huston at NZNOG 2024On Starlink, presented by Geoff Huston at NZNOG 2024
On Starlink, presented by Geoff Huston at NZNOG 2024
 
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
Call Girls Service Chandigarh Lucky ❤️ 7710465962 Independent Call Girls In C...
 
All Time Service Available Call Girls Mg Road 👌 ⏭️ 6378878445
All Time Service Available Call Girls Mg Road 👌 ⏭️ 6378878445All Time Service Available Call Girls Mg Road 👌 ⏭️ 6378878445
All Time Service Available Call Girls Mg Road 👌 ⏭️ 6378878445
 
Enjoy Night⚡Call Girls Dlf City Phase 3 Gurgaon >༒8448380779 Escort Service
Enjoy Night⚡Call Girls Dlf City Phase 3 Gurgaon >༒8448380779 Escort ServiceEnjoy Night⚡Call Girls Dlf City Phase 3 Gurgaon >༒8448380779 Escort Service
Enjoy Night⚡Call Girls Dlf City Phase 3 Gurgaon >༒8448380779 Escort Service
 
AWS Community DAY Albertini-Ellan Cloud Security (1).pptx
AWS Community DAY Albertini-Ellan Cloud Security (1).pptxAWS Community DAY Albertini-Ellan Cloud Security (1).pptx
AWS Community DAY Albertini-Ellan Cloud Security (1).pptx
 
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Pratap Nagar Delhi 💯Call Us 🔝8264348440🔝
 
Real Men Wear Diapers T Shirts sweatshirt
Real Men Wear Diapers T Shirts sweatshirtReal Men Wear Diapers T Shirts sweatshirt
Real Men Wear Diapers T Shirts sweatshirt
 
Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R.Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Green Park Escort Service Delhi N.C.R.
 
(INDIRA) Call Girl Pune Call Now 8250077686 Pune Escorts 24x7
(INDIRA) Call Girl Pune Call Now 8250077686 Pune Escorts 24x7(INDIRA) Call Girl Pune Call Now 8250077686 Pune Escorts 24x7
(INDIRA) Call Girl Pune Call Now 8250077686 Pune Escorts 24x7
 
Russian Call girl in Ajman +971563133746 Ajman Call girl Service
Russian Call girl in Ajman +971563133746 Ajman Call girl ServiceRussian Call girl in Ajman +971563133746 Ajman Call girl Service
Russian Call girl in Ajman +971563133746 Ajman Call girl Service
 
VIP Model Call Girls Hadapsar ( Pune ) Call ON 9905417584 Starting High Prof...
VIP Model Call Girls Hadapsar ( Pune ) Call ON 9905417584 Starting  High Prof...VIP Model Call Girls Hadapsar ( Pune ) Call ON 9905417584 Starting  High Prof...
VIP Model Call Girls Hadapsar ( Pune ) Call ON 9905417584 Starting High Prof...
 
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
Low Sexy Call Girls In Mohali 9053900678 🥵Have Save And Good Place 🥵
 
'Future Evolution of the Internet' delivered by Geoff Huston at Everything Op...
'Future Evolution of the Internet' delivered by Geoff Huston at Everything Op...'Future Evolution of the Internet' delivered by Geoff Huston at Everything Op...
'Future Evolution of the Internet' delivered by Geoff Huston at Everything Op...
 
Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Ind...
Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Ind...Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Ind...
Shikrapur - Call Girls in Pune Neha 8005736733 | 100% Gennuine High Class Ind...
 
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
6.High Profile Call Girls In Punjab +919053900678 Punjab Call GirlHigh Profil...
 
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
Call Girls In Sukhdev Vihar Delhi 💯Call Us 🔝8264348440🔝
 
Call Now ☎ 8264348440 !! Call Girls in Sarai Rohilla Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Sarai Rohilla Escort Service Delhi N.C.R.Call Now ☎ 8264348440 !! Call Girls in Sarai Rohilla Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Sarai Rohilla Escort Service Delhi N.C.R.
 
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRLLucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
Lucknow ❤CALL GIRL 88759*99948 ❤CALL GIRLS IN Lucknow ESCORT SERVICE❤CALL GIRL
 
Call Now ☎ 8264348440 !! Call Girls in Shahpur Jat Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Shahpur Jat Escort Service Delhi N.C.R.Call Now ☎ 8264348440 !! Call Girls in Shahpur Jat Escort Service Delhi N.C.R.
Call Now ☎ 8264348440 !! Call Girls in Shahpur Jat Escort Service Delhi N.C.R.
 

ElasticSearch on AWS

  • 1. ElasticSearch on AWS Continuously Deployed, Immutable and Stateful
  • 2. About me ● Philipp Garbe (@pgarbe) ● Husband ● Father ● AutoScout24 ● Former Microsoft-Fanboy ● Now Docker-Fanboy ;)
  • 3. ● Use case: Logging for AS24 ● Challenge: How to handle the state? ○ Immutable ○ Continuously Deployed ○ Stateful ● Next challenges What to expect
  • 6. STRATEGIC GOALS Goals of the business side ARCHITECTURAL PRINCIPLES High-Level Principles DESIGN AND DELIVERY PRINCIPLES Tactical measures REDUCE TIME TO MARKET Speed, Fast Feedback COST EFFICIENCY Collect metrics to allow decisions cost vs. value. SUPPORT DATA-DRIVEN DECISIONS Listen to users and validate hypothesis. Provide as many relevant metrics & data as possible. YOU BUILT IT, YOU RUN IT The team is responsible for shaping, building, running and maintaining its products. Fast feedback from live and customers helps us to continuously improve. ORGANIZED AROUND BUSINESS CAPABILITIES Build teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver. LOOSELY COUPLED By default avoid sharing and tight coupling, except for the big things in common. Don’t create the next monolith. MACRO AND MICRO ARCHITECTURE Clear separation. Autonomous micro services within the rules and constraints of the macro architecture. AWS FIRST Favor AWS platform service over managed service, over self-hosted OSS, over self-rolled solutions. DATA-DRIVEN / METRIC-DRIVEN Collect metrics from processes and applications. Analyze, alert and act on them. ELIMINATE ACCIDENTAL COMPLEXITY Strive to keep it simple. Focus on essential complexity. You build one, you delete one. AUTONOMOUS TEAMS Make fast local decisions. Be responsible. Know your boundaries. Share findings. INFRASTRUCTURE AS CODE Automate everything: Reproducible, traceable and tested. Immutable servers over snowflake servers. COLLABORATION CULTURE Engineers from all backgrounds work together in collaborative teams as engineers and share responsibilities. No silos. BE BOLD Go into production early. Value monitoring over tests. Recover and learn. Optimize for MTTR not MTBF. SECURITY, COMPLIANCE AND DATA PRIVACY Security must be included from the beginning and everybody’s concern. Keep data-privacy in mind. CONTAINMENT AND BOUNDARIES Align blast radius and vendor lock-in with the boundaries of the organization or business capabilities. Version 1.0 Icons made by Freepik from www.flaticon.com are licensed under CC BY 3.0 https://github.com/autoscout24/scout24-it-principles
  • 7. INFRASTRUCTURE AS CODE Automate everything: Reproducible, traceable and tested. Immutable servers over snowflake servers.
  • 8. Some numbers 12 nodes (m4.4xlarge, EBS gp2 á 2000 GB) 9 indices (one per day) 648 shards 7 Billion docs (new documents peak: 2,500 docs / sec) 16.99 TB of data
  • 9. Challenge: How to handle the state?
  • 11. Continuously Deployed ● One CloudFormation stack per node ● Rolling updates through rake ● Every commit goes to production
  • 12.
  • 14.
  • 15. Monitoring ● CloudWatch ○ Cron-Lambda sends /_cluster/health metrics to CloudWatch ○ CloudWatch alarms based on these metrics ● DataDog ○ Deep Dive