Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017

•

5 likes•745 views

This document summarizes a presentation on chaos engineering at Netflix scale. It discusses how Netflix uses chaos engineering to increase confidence in their systems. Key points include how they define a steady state, implement failure injection and monitoring, prioritize experiments using a criticality score, and record success stories from chaos experiments. The goal of chaos engineering is to run experiments continuously to identify weaknesses before real failures occur.

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Chaos Engineering at Netflix Scale
Nora Jones, Senior Chaos Engineer
@nora_js
DEV334
November 29, 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
KNOWN WAYS TO INCREASE CONFIDENCE

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
UNIT TESTS
Input Output
Component A

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
INTEGRATION TESTS
Input Output
COMPONENT
A
COMPONENT
B
Service C Service D

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHAOS ENGINEERING

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NEW WAY TO INCREASE CONFIDENCE

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHAOS EXPERIMENTS
Service C Service D

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WHY IS THERE A FEAR OF CHAOS WHEN
IT’S INEVITABLE?

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THE “IT DOESN’T APPLY TO ME” FALLACY

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IT APPLIES TO YOU MORE

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCES OF CHAOS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: SOCIALIZATION & MONITORING

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: SOCIALIZATION
Acknowledge
complexity
Define the
steady state

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“I WORK AT A STARTUP, THERE IS NO
STEADY STATE”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TIPS FOR DEFINING STEADY STATE
• Start with non-critical services
• Start in a staging environment, if
possible
• Only include services that want to be
Chaos’ed

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“THE ARMIES OF CHAOS
ARE COMING!”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: SOCIALIZATION
Part of your job as a Chaos Engineer is to
understand the customer and their needs

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: MONITORING
WHAT ARE YOUR KEY BUSINESS METRICS?

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: MONITORING

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SPS: NETFLIX’S KEY BUSINESS METRIC

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 0: MONITORING
DON’T LOSE SIGHT OF YOUR COMPANY’S
CUSTOMERS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 1: GRACEFUL RESTARTS AND
DEGRADATION

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 2: TARGETED CHAOS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 3: CAN WE CAUSE A
CASCADING FAILURE?

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NOT IF THIS FAILS, BUT
WHEN IT FAILS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LATENCY MONKEY: “A LEARNING
OPPORTUNITY”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 4: FAILURE INJECTION

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SAMPLE FAILURE INJECTION LIBRARY

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
NETFLIX FAILURE INJECTION POINTS
HYSTRIX

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service A
Service B
Routing
Failure
injection
Service
Injection
Points
NETFLIX FAILURE INJECTION

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 5: CHAOS AUTOMATION PLATFORM

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service A Service BRouting
100%

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service A
Control
Service A Service BRouting
98%
1%

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service A
Control
Service A
Experiment
Service A Service BRouting
98%
1%
1%

ChAP MONITORING
10:27 10:30 10:33 10:36 10:39 10:42 10:45 10:48

ChAP MONITORING
10:27 10:30 10:33 10:36 10:39 10:42 10:45 10:48
SHORTED

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ChAP GOAL: CHAOS ALL THE THINGS AND
RUN ALL THE TIME

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCE 6: WHAT’S NEXT?

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AUTOMATE EXPERIMENT CREATION AND
PRIORITIZATION

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CRITICIALITY SCORE

RPS Stats Range bucket * number of retries * number of Hystrix Commands =
CRITICALITY SCORE

RPS Stats Range bucket * number of retries * number of Hystrix Commands =
Criticality Score
CRITICALITY SCORE

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FORCES OF CHAOS
0. Socialization and Monitoring
1. Graceful Restarts and Degradation
2. Targeted Chaos
3. Causing a Cascading Failure
4. Failure Injection
5. Chaos Automation Platform

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RECORD CHAOS SUCCESS STORIES

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“We ran a Chaos Experiment that verifies
that our fallback path works and it
successfully caught an issue in the
fallback path before it resulted in an
availability incident”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“While failing calls, we discovered an
increase in requests for the experiment
cluster (even though fallbacks were
successful)…”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“…this likely means whoever was
consuming the fallback was retrying the
call, causing an increase in requests.”

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TAKEAWAYS
• Everyone can and should be doing
Chaos Engineering
• The road to chaos is a learning
opportunity
• Safety is critical, involve your business

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CHAOS DOESN’T CAUSE PROBLEMS.
IT REVEALS THEM.

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
N o r a J o n e s
@ n o r a _ j s

What's hot

In this session, learn how Hulu launched a new Live TV service and Cloud DVR platform using Amazon CloudFront, Amazon S3, Amazon Aurora, and Amazon EC2 to enable the necessary scale for live video ingest, storage, and delivery of content streams. Learn best practices for geographic redundancy, index and naming challenges, hitting performance targets, optimizing for cost, and measuring the user experience to ensure smooth playback for when you build your own video streaming solution.

CTD302_How Hulu reinvented Television using the AWS Cloud

Amazon Web Services

Motion detection triggers have reduced the amount of video recorded by modern devices. But maybe you want to reduce that further—maybe you only care if a car or a person is on-camera before recording or sending a notification. Security cameras and smart doorbells can use Amazon Rekognition to reduce the number of false alarms. Learn how device makers and home enthusiasts are building their own smart layers of person and car detection to reduce false alarms and limit video volume. Learn too how you can use face detection and recognition to notify you when a friend has arrived.

MCL306_Making IoT Smarter with AWS Rekognition.pdf

Amazon Web Services

With increase in popularity of online engagement as a means of entertainment, broad use of wide range of communities have become popular. These communities need to be highly available and resilient at scale. Failure of availability could be fatal to the product that are used by the customer. We will share the process you should use to develop your architectural principles that will allow you to reap the benefits of reduced complexity.

ARC306_High Resiliency & Availability Of Online Entertainment Communities Usi...

Amazon Web Services

End users expect to be able to view static, dynamic, and streaming content anytime, anywhere, and on any device. Amazon CloudFront is a web service that accelerates delivery of your websites, APIs, video content, or other web assets to end users around the globe with low latency, high data transfer speeds, and no commitments. In this session, learn what a content delivery network (CDN) such as Amazon CloudFront is and how it works, the benefits it provides, common challenges and needs, performance, recently released features and examples of how customers are using CloudFront. You will also learn about recustomizing content delivery through AWS Lambda@Edge - a serverless compute service that lets you execute functions to customize the content delivered through CloudFront.

CTD201_Introduction to Amazon CloudFront and AWS Lambda@Edge

Amazon Web Services

Every evening video streaming consumes over 70% of the internet’s bandwidth, with demand only expected to increase as young households forego traditional pay TV for OTT services (whether live, on-demand, ad-supported, transactional, subscription, or a combination thereof). In this session senior tech architects from Netflix, Hulu, and Amazon Video discuss lessons and best practices around hosting largest scale video distribution workloads to enable high traffic consumption at demanding reliability requirements. You will also hear from AWS experts on the latest developments in cloud-based video processing, distribution, and personalization for both live and on-demand streaming.

MAE303-OTT State of Play Innovation at Netflix, Hulu, Amazon Video, and AWS E...

Amazon Web Services

Organizations use application delivery controllers (ADCs) to ensure that their most important applications receive the best performance across their network. In this session, you learn how and why Salesforce used the F5 BIG-IP platform, an ADC solution from AWS Marketplace, during a migration to AWS. To preserve an existing skillset within their business, Salesforce chose AWS Marketplace to first evaluate the solution on the AWS platform before ultimately selecting it as part of their international rollout. You see how BIG-IP performs application routing and security, and how it works with existing AWS networking solutions to provide a consistent experience for domestic and international rollouts. You also learn how Salesforce successfully used the AWS Marketplace Private Offers program to procure an enterprise license and consolidate the expenditure onto their AWS bill.

MSC202_Learn How Salesforce Used ADCs for App Load Balancing for an Internati...

Amazon Web Services

Today, small software teams have the ability to disrupt big markets as more and more businesses start to deliver their products as-a-service. The ability for teams to respond to customers and innovate quickly is their key differentiator. In this session, we will cover how you can begin your DevOps journey by sharing best practices used by the "two pizza" engineering teams at Amazon. We will showcase how you can accelerate developer productivity by implementing continuous Integration and delivery workflows using AWS Developer tools including AWS CodeStar, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline and AWS CodeDeploy. Finally, we will demonstrate how to build an end-to-end CICD pipeline with CodeStar in minutes.

DEV203_Launch Applications the Amazon Way

Amazon Web Services

Whether you’re a cash-strapped startup or an enterprise optimizing spend, it pays to run cost-efficient architectures on AWS. This session reviews a wide range of cost planning, monitoring, and optimization strategies, featuring real-world experience from AWS customers. We cover how to effectively combine Amazon EC2 On-Demand, Reserved, and Spot Instances to handle different use cases; leveraging Auto Scaling to match capacity to workload; and choosing the optimal instance type through load testing. We discuss taking advantage of tiered storage and caching, offloading content to Amazon CloudFront to reduce back-end load, and getting rid of your back end entirely by serverless. Even if you already enjoy the benefits of serverless architectures, we show you how to select the optimal AWS Lambda memory class and how to maximize networking throughput in order to minimize Lambda run-time and therefore execution cost. We also showcase simple tools to help track and manage costs, including Cost Explorer, billing alerts, and AWS Trusted Advisor. This session is your pocket guide for running cost effectively in the AWS Cloud.

ARC303_Running Lean Architectures How to Optimize for Cost Efficiency

Amazon Web Services

Interstella 8888 is an intergalactic trading company that deals in rare resources, but their antiquated monolithic logistics systems are causing the business to lose money. In this workshop, you’ll help Interstella 8888 build a modern microservices-based logistics system to save the company from financial ruin. We’ll give you the hands-on experience you need to run microservices in the real world. Ths includes implementing advanced container scheduling and scaling to deal with variable service requests, implementing a service mesh, issue tracing with Amazon X-Ray, container and instance-level logging with CloudWatch, and load testing. AWS credits are provided. Bring your laptop, and have an active AWS account.

Interstella 8888: Advanced Microservice Operations - CON407 - re:Invent 2017

Amazon Web Services

For organizations looking to glean insights from their data, it is essential to deploy the right environment to successfully support analytics workloads. Learn about the different storage options from AWS, and discuss with our experts how to select the best option for your big data analytics workloads. Hear how one customer, ViaSat, used Amazon EBS for their Apache Kafka and Apache Hadoop workload to improve cost and performance. We also describe best practices and share tips for success throughout.

STG316_Optimizing Storage for Big Data Workloads

Amazon Web Services

What if I told you that you could improve your EC2 performance and availability and save money… Interested? Want to learn how to use all the latest functionality including [NEW] EC2 features launched at re:Invent to optimize your spend… How about now? In this session, you’ll learn how to seamlessly combine On-Demand, Spot and Reserved Instances, and how to use the best practices deployed by customers all over the world for the most common applications and workloads. After just one hour you’ll leave armed with multiple ways to grow your compute capacity and to enable new types of cloud computing applications - without it costing you an arm and a leg.

Optimizing EC2 for Fun and Profit #bigsavings #newfeatures - CMP202 - re:Inve...

Amazon Web Services

Many customers want a disaster recovery environment, and they want to use this environment daily and know that it's in sync with and can support a production workload. This leads them to an active-active architecture. In other cases, users like Netflix and Lyft are distributed over large geographies. In these cases, multi-region active-active deployments are not optional. Designing these architectures is more complicated than it appears, as data being generated at one end needs to be synced with data at the other end. There are also consistency issues to consider. One needs to make trade-off decisions on cost, performance, and consistency. Further complicating matters is the variety of data stores used in the architecture results in a variety replication methods. In this session, we explore how to design an active-active multi-region architecture using AWS services, including Amazon Route 53, Amazon RDS multi-region replication, AWS DMS, and Amazon DynamoDB Streams. We discuss the challenges, trade-offs, and solutions.

ARC319_Multi-Region Active-Active Architecture

Amazon Web Services

As customers put more workloads into AWS, the number of VPCs a customer needs to manage also grows. VPCs can exist across geographically disparate AWS Regions, or run in separate AWS accounts connecting to a common VPC that serves as a global network transit center. This session shows you how to implement a networking construct that AWS calls a transit VPC using the Cisco Cloud Services Router. This network topology simplifies network management and minimizes the number of connections that you need to set up and manage. Even better, it is implemented virtually and doesn't require physical network gear or a physical presence in a colocation transit hub. Come hear why customers are procuring this service through AWS Marketplace and how customers are using the transit VPC for private networking, shared connectivity, and cross-account AWS usage.

How Do I Build a Global Transit Network on AWS? - MSC302 - re:Invent 2017

Amazon Web Services

Learn How AWS is Enabling the World's Most Advanced Media Workflows - CTD202 ...

Amazon Web Services

In this session, we introduce you to the best practices for migrating databases, such as traditional RDBMS or other NoSQL databases to Amazon DynamoDB. We discuss DynamoDB key concepts, evaluation criteria, data modeling in DynamoDB, how to move data into DynamoDB, and data migration key considerations. We share a case study of Samsung Electronics, which migrated their Cassandra cluster to DynamoDB for their Samsung Cloud workload.

DAT320_Moving a Galaxy into Cloud

Amazon Web Services

Analyzing and debugging production distributed applications built using a service oriented, microservices, or serverless architectures is a challenging task. In this session, we introduce AWS X-Ray, an AWS service that makes it easier to identify performance bottlenecks and errors, pinpoint issues to specific services in your application, identify the impact of issues on application users, and visualize the service call graph and the request timelines for your applications. We will also showcase a customer, Chick-fil-A and how they have adopted AWS X-Ray to play a role throughout the microservice lifecycle in order to ensure quality, transparency, and operational visibility for their services on AWS

DEV204_Debugging Modern Applications Introduction to AWS X-Ray

Amazon Web Services

GPUs have a large application in Media and Entertainment workloads. From backend video processing and creation workloads such as VFX/Rendering, transcoding and broadcast playout to high-end creatives as well as video editing workloads. Backed by the NVIDIA Tesla M60 GPUs, G3 instances offer unparalleled power and flexibility to do complex modeling, 3D visualization, computer aided design, seismic visualization, video encoding. G3 instances are the first Amazon EC2 instances to support NVIDIA GRID Virtual Workstation capabilities, with streaming support for four monitors each with up to 4K resolution, and hardware encoding to support up to 10 High Efficiency Video Coding (HEVC) H.265 1080p30 streams or up to 18 H.264 1080p30 streams per GPU for faster video frame processing and improved image fidelity. In this session we will highlight two criticial Media workloads Video Editing via remote application streaming and Broadcast Playout origination from the AWS cloud. We will have Pop Media discuss their remote video editing in the cloud that enables secure remote, real-time editorial and image processing session views. This will be followed by Evertz regarding Discovery Channel’s broadcast Playout application for several live Discovery channels currently.

CMP213_GPU(G3) Applications in Media and Entertainment Workloads

Amazon Web Services

Oracle enterprise applications and middleware such as E-Business Suite, PeopleSoft, Siebel, and WebLogic are central to many IT departments. They often require complex deployments that can greatly benefit from the flexibility, scalability, and security of the cloud. In this session, we discuss architecture patterns and best practices for migrating these applications to and running these applications on AWS. We cover how to work with Oracle enterprise applications and multiple services including Amazon RDS, AWS Database Migration Service, Amazon Elastic File System, and AWS CloudFormation. As part of this, we show examples of successful customer deployments.

Oracle Enterprise Solutions on AWS - ENT326 - re:Invent 2017

Amazon Web Services

The backend for the Snapchat Stories feature includes Snapchat's largest storage write workload. Learn how we rebuilt this workload for Amazon DynamoDB and executed the migration. Safely moving such a critical and high-scale piece of the Stories infrastructure to a new system, right before yearly peak usage, led to interesting challenges. In this session, we cover data model changes to leverage DynamoDB strengths and improve both performance and cost. We also cover challenges and risks in making remote calls across cloud providers, dealing with issues of scale, forecasting capacity requirements, and how to mitigate the risks of taking an unproven system through the dramatic traffic spikes that occur on New Year's Eve.

DAT325_Snapchat Stories

Amazon Web Services

As your application’s infrastructure grows and scales, well-managed container scheduling is critical to ensuring high-availability and resource optimization. In this session, we will deep dive into the challenges and opportunities around container scheduling, as well as the different tools available within Amazon ECS and AWS to carry out efficient container scheduling. We will discuss patterns for container scheduling available with Amazon ECS and the Blox scheduling framework

Deep Dive into Container Scheduling with Amazon ECS - CON404 - re:Invent 2017

Amazon Web Services

What's hot (20)

CTD302_How Hulu reinvented Television using the AWS Cloud

MCL306_Making IoT Smarter with AWS Rekognition.pdf

ARC306_High Resiliency & Availability Of Online Entertainment Communities Usi...

CTD201_Introduction to Amazon CloudFront and AWS Lambda@Edge

MAE303-OTT State of Play Innovation at Netflix, Hulu, Amazon Video, and AWS E...

MSC202_Learn How Salesforce Used ADCs for App Load Balancing for an Internati...

DEV203_Launch Applications the Amazon Way

ARC303_Running Lean Architectures How to Optimize for Cost Efficiency

Interstella 8888: Advanced Microservice Operations - CON407 - re:Invent 2017

STG316_Optimizing Storage for Big Data Workloads

Optimizing EC2 for Fun and Profit #bigsavings #newfeatures - CMP202 - re:Inve...

ARC319_Multi-Region Active-Active Architecture

How Do I Build a Global Transit Network on AWS? - MSC302 - re:Invent 2017

Learn How AWS is Enabling the World's Most Advanced Media Workflows - CTD202 ...

DAT320_Moving a Galaxy into Cloud

DEV204_Debugging Modern Applications Introduction to AWS X-Ray

CMP213_GPU(G3) Applications in Media and Entertainment Workloads

Oracle Enterprise Solutions on AWS - ENT326 - re:Invent 2017

DAT325_Snapchat Stories

Deep Dive into Container Scheduling with Amazon ECS - CON404 - re:Invent 2017

Similar to Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017

10 Lessons from 10 Years of AWS

Adrian Hornsby

Launch Applications the Amazon Way - AWS Online Tech Talks

Amazon Web Services

With the rise of micro-services and large-scale distributed architectures, software systems have grown increasingly complex and hard to understand. Adding to that complexity, the velocity of software delivery has also dramatically increased, resulting in failures being harder to predict and contain. While the cloud allows for high availability, redundancy and fault-tolerance, no single component can guarantee 100% uptime. Therefore, we have to understand availability but especially learn how to design architectures with failure in mind. And since failures have become more and more chaotic in nature, we must turn to chaos engineering in order to identify failures before they become outages. In this talk, I will deep dive into availability, reliability and large-scale architectures and make an introduction to chaos engineering, a discipline that promotes breaking things on purpose in order to learn how to build more resilient systems.

Chaos Engineering: Why Breaking Things Should Be Practised.

Adrian Hornsby

Governing cloud infrastructure at scale requires software that enables you to capture and drive management from internal policies, best practices, and reference architectures. A policy-driven management and governance strategy is critical to successfully operate in cloud and hybrid environments. As infrastructure grows, you might leverage knowledge that extends beyond the organization. An open-source “cloud policy framework” enables users to leverage a community that can help define and tune best practice policies, and help SaaS vendors and ISVs capture the best way to manage an application and share it with customers. A well-defined management and governance strategy enables you to put automation in place that keeps your cloud running securely and efficiently without having to take it on as a full-time job. This session discusses the development of a “cloud policy framework” that enables users to leverage open source rule definition organizations can use to govern their cloud. Learn best practice policies for managing all aspects of services, applications, and infrastructure across cost, availability, performance, security and usage. Session sponsored by CloudHealth Technologies

Leveraging a Cloud Policy Framework - From Zero to Well Governed - ENT318 - r...

Amazon Web Services

AWS re:Invent 2017 | CloudHealth Tech Session

CloudHealth by VMware

Presented by Kevin Downs, Senior AWS Technical Partner Manager of New Relic, at Amazon Web Services re:Invent 2017. Static applications living on long running servers are becoming history in the cloud along with the monitoring assumptions we used to make. Now, we routinely deploy a range of services from autoscaling compute to decoupled message queues to serverless. These dynamic services, along with microservice architectures both break and need to co-exist with traditional monitoring and instrumentation approaches. Whether you are building new apps, were told to migrate yesterday, are currently migrating, or are already scaling your apps on AWS, this session will dive into the how, where, and when to monitor your applications and infrastructure, no matter where your apps run. You’ll hear best practices we've learned from our customers and running our own service (1.5 Billion+ metrics per minute) too. Join us for a little bit of history and a whole lot of now as we show you how and what you need to scale and prove your success in the cloud.

DEV209 A Field Guide to Monitoring in the Cloud: From Lift and Shift to AWS L...

New Relic

The AWS Philosophy of Security - SID322 - re:Invent 2017

Amazon Web Services

As containers become more embedded in the platform tools, debug tools, traces and logs become increasingly important. Nare Hayrapetyan, Senior Software Engineer and Calvin French-Owen, Senior Technical Officer for Segment will discuss the principals of monitoring and debugging containers and the tools Segment has implemented and built for logging, alerting, metric collection, and debugging of containerized services running on Amazon ECS.

CON320_Monitoring, Logging and Debugging Containerized Services

Amazon Web Services

Cloud is the new normal - it continues to deliver amazing new technologies and drive us to innovate our operational models. Cloud also gives us new capabilities to see the tremendous financial impact to our businesses. As you journey into the cloud, your teams will need to wield the cloud power responsibly. They will need more visibility into the financial impact of their resource usage in order to make the best decisions for today and for the longer term. Join Cloudability and HERE Technologies as we guide you through the journey to financial agility on the cloud as we have seen it play it out in hundreds of customers. You’ll walk away with the steps you need to take and the mindset your teams will need to adopt in order to achieve financial agility. Topics include: - Discovering and validating your goals (needs, requirements, responsibilities) - Analyzing and closing the gaps between your goals and your reality - Developing a new operational model that provides the financial agility and predictability - HERE Technologies case study Session Sponsored by Cloudability

From Cloud Cost Management to Financial Agility: The Journey to Success - ENT...

Amazon Web Services

This session walks you through some of the more advanced features offered in Alexa Skill Builder, like Dialog Management, Entity Resolution, state management, session persistence, and maintaining context. Using Dialog Management, you can engage skill users in a multi-turn dialog to elicit and confirm slots for an intent. Using Entity Resolution, you can greatly simplify slot management by mapping multiple synonyms of your slot to a unique ID. We couple these conversational techniques with the management of session state and persistence to enable memory and personalization.

Conversation and Memory - ALX401-R - re:Invent 2017

Amazon Web Services

Keynote - AWSome Day Copenhagen

Amazon Web Services

AWSome Day - Opening Keynote

Adrian Hornsby

How serverless helps startups innovate and scale

Gabe Hollombe

When you use the cloud to enable speed and agility, how do you know if you did it right? We are on a mission to help builders follow industry best practices within security guide rails by creating the largest compliance-as-code repo, available to all. Compliance-as-code is the idea to translate those best practices, guide rails, policies, or standards into codified unit testing. Apply this to your AWS environment to provide insights on what can/must be improved. Learn why compliance-as-code matters to gain speed (by getting developers, architects, and security pros on the same page), how it is currently used (demo), and how to start to use it or be part of building it.

Building the Largest Repo for Serverless Compliance-as-Code - SID205 - re:Inv...

Amazon Web Services

Digital transformation means something new every decade. This sort of constant change is stressful for any organization, but especially for the large enterprise. Because while nearly every enterprise began as a small, scrappy team, success means growth, and growth means accumulation—more technology, more processes, more people, and more red tape. In this session, we’ll give an executive-level view of how the cloud removes the stressful burden of on-premise applications, legacy technology, and outdated processes from the enterprise. We’ll examine what digital transformation really means, and offer prescriptive guidance around decluttering the enterprise, security, culture, and more. AWS enables every enterprise to invest in strategic innovation and modernization. With the AWS cloud, enterprises become more agile, more efficient, and are better equipped to keep pace with constant change. As a partner, AWS enables every enterprise to live its own version of the Amazon Story—one of growth without burdensome accumulation, success without slowness, and constant innovation.

Keynote: What Transformation Really Means for the Enterprise - AWS Transforma...

Amazon Web Services

Fed up with stop and go in your data center? Shift into overdrive and pull into the fast lane! Learn how AutoScout24, the largest online car marketplace Europe-wide, is building its Autobahn in the Cloud. The secret ingredient? Culture! Because “cloud” is only half of the digital transformation story. The other half is how your organization deals with cultural change as you transition from the old world of IT into building microservices on AWS, with agile DevOps teams in a true „you build it, you run it“ fashion. Listen to stories from the trenches, powered by Amazon Kinesis, Amazon DynamoDB, AWS Lambda, Amazon ECS, Amazon API Gateway and much more, backed by AWS Partners, AWS Professional Services, and AWS Enterprise Support. Learn how to become cloud native, evolve your architecture, drive cultural change across teams, and manage your company’s transformation for the future.

The Enterprise Fast Lane - What Your Competition Doesn't Want You to Know abo...

Amazon Web Services

Pharmaceutical company processes tend to be slow when dealing with customer-facing applications that contain FDA-validated messages, all while maintaining infrastructure and security standards. In this session, discover how Mylan, a US–based global generic and specialty pharmaceutical company, overcame these obstacles and provided scalable solutions by leveraging AWS DevOps methods that lower time to market, while maintaining robust security and release management practices. During the presentation, learn how Mylan redefined process models such as infrastructure change management to define new security and process models. Additionally, learn how Mylan used services like Amazon S3, Elastic Load Balancing (ELB), and AWS CloudFormation to define these new models.

LFS307_Using AWS to Maximize Digital Marketing Reach and Efficiency

Amazon Web Services

Join this State of the Union to learn about the latest developments from Amazon for enterprise workloads such as Windows, VMware, and SAP. Sandy Carter, AWS vice president for Enterprise Workloads, discusses the evolution of AWS services for enterprise workloads and the new features and services that we are launching for Windows and VMware. She shares the company’s vision for continuing to innovate in this space to make AWS the premier place for enterprise customers. Also, several major customers discuss their own experience running enterprise workloads on AWS as well as pursuing new solutions in areas like AI and IoT.

Enterprise Innovation? Yes, with AWS Cloud, AI, and IoT - WIN201 - re:Invent ...

Amazon Web Services

For many organizations, a perceived lack of cloud skills in their staff can limit their move to the cloud. Proper training of your engineers and developers can speed the pace of adoption, cloud migration, and delivery of business benefits by effectively operating the AWS Cloud. In this session, we discuss field-proven, prescriptive steps for reskilling and scaling your technical teams so that you can use the AWS Cloud securely, efficiently, and effectively. Speaker: Jonathan Allen, EMEA Enterprise Strategist & Evangelist, Amazon Web Services

How to get from Zero to Hundreds of Certified Engineers

Amazon Web Services

We'll show you how to take your application and launch it quickly on a variety of AWS infrastructure. You'll learn how to leverage CodeStar, CodeBuild, CodeDeploy, and Cloud9 to provide your startup with reliable, flexible, and cost efficient build pipelines in minutes. This will set your technical teams up for faster deploys and consistent development environments allowing you to focus on your product, not your deployment process. This is a key pain point for early stage startups, learn how to solve it before it starts to impact your team's productivity.

Launching applications the Amazon Way

Amazon Web Services

Similar to Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017 (20)

10 Lessons from 10 Years of AWS

Launch Applications the Amazon Way - AWS Online Tech Talks

Chaos Engineering: Why Breaking Things Should Be Practised.

Leveraging a Cloud Policy Framework - From Zero to Well Governed - ENT318 - r...

AWS re:Invent 2017 | CloudHealth Tech Session

DEV209 A Field Guide to Monitoring in the Cloud: From Lift and Shift to AWS L...

The AWS Philosophy of Security - SID322 - re:Invent 2017

CON320_Monitoring, Logging and Debugging Containerized Services

From Cloud Cost Management to Financial Agility: The Journey to Success - ENT...

Conversation and Memory - ALX401-R - re:Invent 2017

Keynote - AWSome Day Copenhagen

AWSome Day - Opening Keynote

How serverless helps startups innovate and scale

Building the Largest Repo for Serverless Compliance-as-Code - SID205 - re:Inv...

Keynote: What Transformation Really Means for the Enterprise - AWS Transforma...

The Enterprise Fast Lane - What Your Competition Doesn't Want You to Know abo...

LFS307_Using AWS to Maximize Digital Marketing Reach and Efficiency

Enterprise Innovation? Yes, with AWS Cloud, AI, and IoT - WIN201 - re:Invent ...

How to get from Zero to Hundreds of Certified Engineers

Launching applications the Amazon Way

More from Amazon Web Services

Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS. In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Amazon Web Services

La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup. Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti. Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Amazon Web Services

Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi

Esegui pod serverless con Amazon EKS e AWS Fargate

Amazon Web Services

Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.

Costruire Applicazioni Moderne con AWS

Amazon Web Services

L’utilizzo dei container è in continua crescita. Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili. I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!

Come spendere fino al 90% in meno con i container e le istanze spot

Amazon Web Services

In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th. Event Agenda : Open banking so far (short recap) • PSD2, OB UK, OB Australia, OB LATAM, OB Israel Intro to Open Finance marketplace • Scope • Features • Tech overview and Demo The role of the Cloud The Future of APIs • Complying with regulation • Monetizing data / APIs • Business models • Time to market One platform for all: a Strategic approach Q&A

Open banking as a service

Amazon Web Services

Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc. AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta. Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

Amazon Web Services

Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity. AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet. Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Amazon Web Services

Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Amazon Web Services

Computer Vision con AWS

Amazon Web Services

Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti. Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.

Database Oracle e VMware Cloud on AWS i miti da sfatare

Amazon Web Services

Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti. Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire. Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito. In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

Amazon Web Services

Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline. Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.

API moderne real-time per applicazioni mobili e web

Amazon Web Services

Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi. La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali. In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Amazon Web Services

Tools for building your MVP on AWS

Amazon Web Services

How to Build a Winning Pitch Deck

Amazon Web Services

Building a web application without servers

Amazon Web Services

Fundraising Essentials

Amazon Web Services

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Amazon Web Services

Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.

Introduzione a Amazon Elastic Container Service

Amazon Web Services

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...

Big Data per le Startup: come creare applicazioni Big Data in modalità Server...

Esegui pod serverless con Amazon EKS e AWS Fargate

Costruire Applicazioni Moderne con AWS

Come spendere fino al 90% in meno con i container e le istanze spot

Open banking as a service

Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...

OpsWorks Configuration Management: automatizza la gestione e i deployment del...

Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads

Computer Vision con AWS

Database Oracle e VMware Cloud on AWS i miti da sfatare

Crea la tua prima serverless ledger-based app con QLDB e NodeJS

API moderne real-time per applicazioni mobili e web

Database Oracle e VMware Cloud™ on AWS: i miti da sfatare

Tools for building your MVP on AWS

How to Build a Winning Pitch Deck

Building a web application without servers

Fundraising Essentials

AWS_HK_StartupDay_Building Interactive websites while automating for efficien...

Introduzione a Amazon Elastic Container Service

Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017

14.

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TIPS FOR DEFINING STEADY STATE • Start with non-critical services • Start in a staging environment, if possible • Only include services that want to be Chaos’ed

39.

40.

41.

42.

43.

44.

45. TYPES OF CHAOS FAILURES

46. TYPES OF CHAOS FAILURES

47. Criteria&API

48.

49.

58. ChAP MONITORING 10:27 10:30 10:33 10:36 10:39 10:42 10:45 10:48

59. ChAP MONITORING 10:27 10:30 10:33 10:36 10:39 10:42 10:45 10:48 SHORTED

64.

65.

66. ChAP’S MONOCLE

67. ChAP’S MONOCLE

68. ChAP’S MONOCLE

71. RPS Stats Range bucket * number of retries * number of Hystrix Commands = CRITICALITY SCORE

72. RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality Score CRITICALITY SCORE

73. RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality Score CRITICALITY SCORE

74. RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality Score CRITICALITY SCORE

75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FORCES OF CHAOS 0. Socialization and Monitoring 1. Graceful Restarts and Degradation 2. Targeted Chaos 3. Causing a Cascading Failure 4. Failure Injection 5. Chaos Automation Platform

77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “We ran a Chaos Experiment that verifies that our fallback path works and it successfully caught an issue in the fallback path before it resulted in an availability incident”

78.

79.

80.

81.

82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “While failing calls, we discovered an increase in requests for the experiment cluster (even though fallbacks were successful)…”

85. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TAKEAWAYS • Everyone can and should be doing Chaos Engineering • The road to chaos is a learning opportunity • Safety is critical, involve your business

Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017

Similar to Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Performing Chaos at Netflix Scale - DEV334 - re:Invent 2017