SlideShare a Scribd company logo
1 of 69
Download to read offline
Don’t let this happen to you!
          Cloud, complexity and drift

                  James Urquhart
@jamesurquhart http://gigaom.com/cloud
May 6, 2010
Don't let this happen to you! Cloud, complexity and drift
”[A] large fundamental trader chose to execute [a
$4.1B] sell program via an automated execution
algorithm ('Sell Algorithm')."


              - Findings Regarding The Market Events of May 6, 2010
                  http://www.sec.gov/news/studies/2010/marketevents-report.pdf
"...the Sell Algorithm…executed the sell program extremely
rapidly in just 20 minutes.”

The market responded, and trading volume increased…

"... [The Sell Algorithm] responded to the increased volume by
increasing the rate at which it was feeding the orders into the
market."
                 - Findings Regarding The Market Events of May 6, 2010
                     http://www.sec.gov/news/studies/2010/marketevents-report.pdf
Don't let this happen to you! Cloud, complexity and drift
Quick!
What was the root cause?
Automatic
  Trading     Market A
Algorithm 1
Automatic
  Trading             Market A
Algorithm 1



       Automatic     Automatic     Automatic
         Trading       Trading       Trading
       Algorithm 2   Algorithm 3   Algorithm 4
Market B
Automatic
  Trading             Market A
Algorithm 1



       Automatic     Automatic     Automatic
         Trading       Trading       Trading
       Algorithm 2   Algorithm 3   Algorithm 4
Automatic
                         Automatic                     Trading
                           Trading                   Algorithm 6
                         Algorithm 5



                                                      Market B
Automatic
  Trading             Market A
Algorithm 1



       Automatic     Automatic         Automatic
         Trading       Trading           Trading
       Algorithm 2   Algorithm 3       Algorithm 4
Automatic
                         Automatic                     Trading
                           Trading                   Algorithm 6
                         Algorithm 5



                                                      Market B
Automatic
  Trading             Market A
Algorithm 1



       Automatic     Automatic         Automatic
         Trading       Trading           Trading
       Algorithm 2   Algorithm 3       Algorithm 4
Automatic
                           Automatic                     Trading
                             Trading                   Algorithm 6
                           Algorithm 5


 Automatic
  Automatic
  Trading                                               Market B
   Automatic
   Trading
Algorithm N
    Trading             Market A
 Algorithm N
  Algorithm N



         Automatic     Automatic         Automatic
           Trading       Trading           Trading
         Algorithm 2   Algorithm 3       Algorithm 4
Don't let this happen to you! Cloud, complexity and drift
“Root cause” is not an
 answer—it’s a clue.
"May 6 was…an important reminder of the
interconnectedness of our derivatives and
securities markets, particularly with respect to
index products."


             - Findings Regarding The Market Events of May 6, 2010
                 http://www.sec.gov/news/studies/2010/marketevents-report.pdf
Interconnectedness
The Internet is about
interconnectedness.
APIs are about
interconnectedness.
“The cloud” is about
interconnectedness.
Don't let this happen to you! Cloud, complexity and drift
Both cloud computing
and stock markets are
  complex adaptive
       systems.
COMPLEX?!?
Don't let this happen to you! Cloud, complexity and drift
What are
Complex Adaptive
   Systems?
Don't let this happen to you! Cloud, complexity and drift
Don't let this happen to you! Cloud, complexity and drift
Don't let this happen to you! Cloud, complexity and drift
A large number of individual agents
Agents

Other    Information/A
             ctions
                          State
Agents
                         Learning
                                    Information/A   Other
                          Rules         ctions
                                                    Agents
A large number of individual agents

+ dynamic interactions between agents
Dynamic Interaction




     From NetLogo 4.1.3
A large number of individual agents

+ dynamic interactions between agents

+ rules for reacting to/interacting with
other agents
Rules
Equals
A system that:

demonstrates emergent behavior,
Emergent Behavior
A system that:

demonstrates emergent behavior,

can be modeled,
Models

             Time




Flocking model from
NetLogo 4.1.3
http://ccl.northwestern.edu/netlogo/
A system that:

demonstrates emergent behavior,

can be modeled,

but

makes precise prediction of future
behavior impossible.
Unpredictability
Unpredictability
Unpredictability
Unpredictability
So…what can you do?
EMBRACE
COMPLEXITY
Never!
EMBRACE
COMPLEXITY
 SYSTEMS
Embrace Systems
   Thinking
Practical advice?
Do your homework!
Don't let this happen to you! Cloud, complexity and drift
Design for resilience
“The goal of producing a maximum
sustained yield may result in a more stable
system of reduced resilience.”



   “Command and Control and the Pathology of Natural Resource Management”,
   C. S. Holling and Gary K. Meffe
   http://landscape.forest.wisc.edu/courses/Landscape565spr01/Holling_Meffe1996.pdf
“[T]he resilience-stability tradeoff is more
than just a simple transformation in
distribution. …[A]gents adapt to a
prolonged period of stability in such a
manner that the system cannot ‘withstand
even modest adverse shocks.’”

   “The Euro and the Resilience-Stability Tradeoff”, Ashwin Parameswaran
   http://www.macroresilience.com/2011/11/14/the-euro-and-the-resilience-stability-tradeoff/
Don't let this happen to you! Cloud, complexity and drift
Focus on relationships
Automatic
                           Automatic                     Trading
                             Trading                   Algorithm 6
                           Algorithm 5


 Automatic
  Automatic
  Trading                                               Market B
   Automatic
   Trading
Algorithm N
    Trading             Market A
 Algorithm N
  Algorithm N



         Automatic     Automatic         Automatic
           Trading       Trading           Trading
         Algorithm 2   Algorithm 3       Algorithm 4
Automatic
                           Automatic                     Trading
                             Trading                   Algorithm 6
                           Algorithm 5


 Automatic
  Automatic
  Trading                                               Market B
   Automatic
   Trading
Algorithm N
    Trading             Market A
 Algorithm N
  Algorithm N



         Automatic     Automatic         Automatic
           Trading       Trading           Trading
         Algorithm 2   Algorithm 3       Algorithm 4
Circuit Breaker
http://techblog.netflix.com/2011/12/makin
g-netflix-api-more-resilient.html
Avoid Drift
Drift
Avoiding Drift:
Test production
Release the
monkeys!!!
•   Failure
•   Degredation
•   Usage
Avoiding Drift:
The agent is not the
      system
App-centric              System-centric
• Monitor each app       • Monitor system at many
  separately               levels, and use as feedback
• Dig for root cause     • Search for system weakness
• Attempt to stabilize   • Focus on resilience
Avoiding Drift:
        YOU
are part of the system
Don’t let this happen to you!
    Cloud, complexity and drift

More Related Content

What's hot

Basics of Java Concurrency
Basics of Java ConcurrencyBasics of Java Concurrency
Basics of Java Concurrencykshanth2101
 
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...Christopher Frohoff
 
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdNavigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdChristian Posta
 
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012Scott Sutherland
 
DNS exfiltration using sqlmap
DNS exfiltration using sqlmapDNS exfiltration using sqlmap
DNS exfiltration using sqlmapMiroslav Stampar
 
Networking in Java with NIO and Netty
Networking in Java with NIO and NettyNetworking in Java with NIO and Netty
Networking in Java with NIO and NettyConstantine Slisenka
 
Microservice API Gateways with NGINX
Microservice API Gateways with NGINXMicroservice API Gateways with NGINX
Microservice API Gateways with NGINXGeoffrey Filippi
 
Windows Operating System Archaeology
Windows Operating System ArchaeologyWindows Operating System Archaeology
Windows Operating System Archaeologyenigma0x3
 
Exploiting Deserialization Vulnerabilities in Java
Exploiting Deserialization Vulnerabilities in JavaExploiting Deserialization Vulnerabilities in Java
Exploiting Deserialization Vulnerabilities in JavaCODE WHITE GmbH
 
Bootstrap for Beginners
Bootstrap for BeginnersBootstrap for Beginners
Bootstrap for BeginnersD'arce Hess
 
Modelo Integración Continua en entornos de QA
Modelo Integración Continua en entornos de QAModelo Integración Continua en entornos de QA
Modelo Integración Continua en entornos de QAMkt Manager Data Adviser
 
The hardest part of microservices: your data
The hardest part of microservices: your dataThe hardest part of microservices: your data
The hardest part of microservices: your dataChristian Posta
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Chris Richardson
 

What's hot (20)

AEM 6.X (With Basics) Training Syllabus
AEM 6.X (With Basics) Training SyllabusAEM 6.X (With Basics) Training Syllabus
AEM 6.X (With Basics) Training Syllabus
 
Basics of Java Concurrency
Basics of Java ConcurrencyBasics of Java Concurrency
Basics of Java Concurrency
 
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...
OWASP SD: Deserialize My Shorts: Or How I Learned To Start Worrying and Hate ...
 
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and LinkerdNavigating the service mesh landscape with Istio, Consul Connect, and Linkerd
Navigating the service mesh landscape with Istio, Consul Connect, and Linkerd
 
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012
SQL Server Exploitation, Escalation, Pilfering - AppSec USA 2012
 
Codeigniter
CodeigniterCodeigniter
Codeigniter
 
DNS exfiltration using sqlmap
DNS exfiltration using sqlmapDNS exfiltration using sqlmap
DNS exfiltration using sqlmap
 
Networking in Java with NIO and Netty
Networking in Java with NIO and NettyNetworking in Java with NIO and Netty
Networking in Java with NIO and Netty
 
Microservice API Gateways with NGINX
Microservice API Gateways with NGINXMicroservice API Gateways with NGINX
Microservice API Gateways with NGINX
 
Windows Operating System Archaeology
Windows Operating System ArchaeologyWindows Operating System Archaeology
Windows Operating System Archaeology
 
Exploiting Deserialization Vulnerabilities in Java
Exploiting Deserialization Vulnerabilities in JavaExploiting Deserialization Vulnerabilities in Java
Exploiting Deserialization Vulnerabilities in Java
 
Solid principles
Solid principlesSolid principles
Solid principles
 
Bootstrap for Beginners
Bootstrap for BeginnersBootstrap for Beginners
Bootstrap for Beginners
 
Bootstrap 3 vs. bootstrap 4
Bootstrap 3 vs. bootstrap 4Bootstrap 3 vs. bootstrap 4
Bootstrap 3 vs. bootstrap 4
 
Asp.net mvc 5 ppt
Asp.net mvc 5 pptAsp.net mvc 5 ppt
Asp.net mvc 5 ppt
 
Xss (cross site scripting)
Xss (cross site scripting)Xss (cross site scripting)
Xss (cross site scripting)
 
Spring Security 5
Spring Security 5Spring Security 5
Spring Security 5
 
Modelo Integración Continua en entornos de QA
Modelo Integración Continua en entornos de QAModelo Integración Continua en entornos de QA
Modelo Integración Continua en entornos de QA
 
The hardest part of microservices: your data
The hardest part of microservices: your dataThe hardest part of microservices: your data
The hardest part of microservices: your data
 
Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!Dark Energy, Dark Matter and the Microservices Patterns?!
Dark Energy, Dark Matter and the Microservices Patterns?!
 

Viewers also liked

Look ma, No Frameworks - JBcnConf 2015
Look ma, No Frameworks - JBcnConf 2015Look ma, No Frameworks - JBcnConf 2015
Look ma, No Frameworks - JBcnConf 2015Pablo Chacin
 
Shit My Cloud Evangelist Says...Just Not To My CSO
Shit My Cloud Evangelist Says...Just Not To My CSOShit My Cloud Evangelist Says...Just Not To My CSO
Shit My Cloud Evangelist Says...Just Not To My CSOChristofer Hoff
 
The Cloudcast - Krispy Kreme Challenge 2015
The Cloudcast - Krispy Kreme Challenge 2015The Cloudcast - Krispy Kreme Challenge 2015
The Cloudcast - Krispy Kreme Challenge 2015thecloudcastnet
 
Chef for OpenStack - OpenStack Fall 2012 Summit
Chef for OpenStack  - OpenStack Fall 2012 SummitChef for OpenStack  - OpenStack Fall 2012 Summit
Chef for OpenStack - OpenStack Fall 2012 SummitMatt Ray
 
Cloud Computing is not simple - The complexity is in the details
Cloud Computing is not simple - The complexity is in the detailsCloud Computing is not simple - The complexity is in the details
Cloud Computing is not simple - The complexity is in the detailsRene Buest
 
Social Media Optimization (Diplomarbeit)
Social Media Optimization (Diplomarbeit)Social Media Optimization (Diplomarbeit)
Social Media Optimization (Diplomarbeit)D. Lewicki
 
Krispy kreme doughtnuts (kkd)
Krispy kreme doughtnuts (kkd)Krispy kreme doughtnuts (kkd)
Krispy kreme doughtnuts (kkd)wajahathailian
 
Krispy cream case study final
Krispy cream  case study   finalKrispy cream  case study   final
Krispy cream case study finalwajahathailian
 
Подобряване на ефективността на регионалната структура на националния статист...
Подобряване на ефективността на регионалната структура на националния статист...Подобряване на ефективността на регионалната структура на националния статист...
Подобряване на ефективността на регионалната структура на националния статист...Светла Иванова
 
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...Maurizio De Filippis
 
Presentation STEM careers on 9th Math International Conference
Presentation STEM careers on 9th Math International ConferencePresentation STEM careers on 9th Math International Conference
Presentation STEM careers on 9th Math International ConferencePanagiota Argiri
 
Bang portfolio
Bang portfolioBang portfolio
Bang portfolioAine Doris
 

Viewers also liked (15)

Look ma, No Frameworks - JBcnConf 2015
Look ma, No Frameworks - JBcnConf 2015Look ma, No Frameworks - JBcnConf 2015
Look ma, No Frameworks - JBcnConf 2015
 
Shit My Cloud Evangelist Says...Just Not To My CSO
Shit My Cloud Evangelist Says...Just Not To My CSOShit My Cloud Evangelist Says...Just Not To My CSO
Shit My Cloud Evangelist Says...Just Not To My CSO
 
The Cloudcast - Krispy Kreme Challenge 2015
The Cloudcast - Krispy Kreme Challenge 2015The Cloudcast - Krispy Kreme Challenge 2015
The Cloudcast - Krispy Kreme Challenge 2015
 
Chef for OpenStack - OpenStack Fall 2012 Summit
Chef for OpenStack  - OpenStack Fall 2012 SummitChef for OpenStack  - OpenStack Fall 2012 Summit
Chef for OpenStack - OpenStack Fall 2012 Summit
 
Cloud Computing is not simple - The complexity is in the details
Cloud Computing is not simple - The complexity is in the detailsCloud Computing is not simple - The complexity is in the details
Cloud Computing is not simple - The complexity is in the details
 
Social Media Optimization (Diplomarbeit)
Social Media Optimization (Diplomarbeit)Social Media Optimization (Diplomarbeit)
Social Media Optimization (Diplomarbeit)
 
Krispy kreme doughtnuts (kkd)
Krispy kreme doughtnuts (kkd)Krispy kreme doughtnuts (kkd)
Krispy kreme doughtnuts (kkd)
 
Krispy cream case study final
Krispy cream  case study   finalKrispy cream  case study   final
Krispy cream case study final
 
Q and A - Rav Dhillon Argyle Coins
Q and A - Rav Dhillon Argyle CoinsQ and A - Rav Dhillon Argyle Coins
Q and A - Rav Dhillon Argyle Coins
 
Подобряване на ефективността на регионалната структура на националния статист...
Подобряване на ефективността на регионалната структура на националния статист...Подобряване на ефективността на регионалната структура на националния статист...
Подобряване на ефективността на регионалната структура на националния статист...
 
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...
Allora Blu & Gli amici di Davide il drago ONLUS: "Nati per vivere" di M. Jank...
 
New marketer breed
New marketer breed New marketer breed
New marketer breed
 
College mediacoaches bibliotheken 200317
College mediacoaches bibliotheken   200317College mediacoaches bibliotheken   200317
College mediacoaches bibliotheken 200317
 
Presentation STEM careers on 9th Math International Conference
Presentation STEM careers on 9th Math International ConferencePresentation STEM careers on 9th Math International Conference
Presentation STEM careers on 9th Math International Conference
 
Bang portfolio
Bang portfolioBang portfolio
Bang portfolio
 

Similar to Don't let this happen to you! Cloud, complexity and drift

Algorithmic Trading-An Introduction
Algorithmic Trading-An IntroductionAlgorithmic Trading-An Introduction
Algorithmic Trading-An IntroductionRajeev Ranjan
 
Introduction to Algoaction -Web Based Trading Platform
Introduction to Algoaction -Web Based Trading PlatformIntroduction to Algoaction -Web Based Trading Platform
Introduction to Algoaction -Web Based Trading PlatformFinfolab Technologies
 
A case for intelligent autonomous ai (iai)
A case for intelligent autonomous ai (iai)A case for intelligent autonomous ai (iai)
A case for intelligent autonomous ai (iai)Mark Albala
 
List of features for harmonic pattern indicator
List of features for harmonic pattern indicatorList of features for harmonic pattern indicator
List of features for harmonic pattern indicatorLeadingTrader21
 
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptx
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptxThe Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptx
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptxShubhamKumar758510
 
Harmonic pattern scanner mt5
Harmonic pattern scanner mt5Harmonic pattern scanner mt5
Harmonic pattern scanner mt5LeadingTrader21
 
Features Enhance trades & maximize gains
Features Enhance trades & maximize gainsFeatures Enhance trades & maximize gains
Features Enhance trades & maximize gainswoodenresort39
 
Winning trading logic – detecting supply and demand unbalance
Winning trading logic – detecting supply and demand unbalanceWinning trading logic – detecting supply and demand unbalance
Winning trading logic – detecting supply and demand unbalanceLeadingTrader21
 
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdf
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdfAdding Peak Trough Analysis Tool in your Chart – MetaTrader.pdf
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdfLeadingTrader21
 
Quantopian: Crowd-sourced Systematic Alpha
Quantopian: Crowd-sourced Systematic AlphaQuantopian: Crowd-sourced Systematic Alpha
Quantopian: Crowd-sourced Systematic AlphaQuantopian
 
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017MLconf
 
Machine Learning and RoboCop Testing
Machine Learning and RoboCop TestingMachine Learning and RoboCop Testing
Machine Learning and RoboCop TestingIosif Itkin
 
Algorithmic Trading: an Overview
Algorithmic Trading: an Overview Algorithmic Trading: an Overview
Algorithmic Trading: an Overview EXANTE
 
Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1LeadingTrader21
 
Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1LeadingTrader21
 
Grid Forex Strategies
Grid Forex StrategiesGrid Forex Strategies
Grid Forex StrategiesHenry Foster
 

Similar to Don't let this happen to you! Cloud, complexity and drift (20)

Algorithmic Trading-An Introduction
Algorithmic Trading-An IntroductionAlgorithmic Trading-An Introduction
Algorithmic Trading-An Introduction
 
Introduction to Algoaction -Web Based Trading Platform
Introduction to Algoaction -Web Based Trading PlatformIntroduction to Algoaction -Web Based Trading Platform
Introduction to Algoaction -Web Based Trading Platform
 
A case for intelligent autonomous ai (iai)
A case for intelligent autonomous ai (iai)A case for intelligent autonomous ai (iai)
A case for intelligent autonomous ai (iai)
 
List of features for harmonic pattern indicator
List of features for harmonic pattern indicatorList of features for harmonic pattern indicator
List of features for harmonic pattern indicator
 
Algorithmic trading
Algorithmic tradingAlgorithmic trading
Algorithmic trading
 
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptx
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptxThe Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptx
The Power of AI Trading Bots_ How They Are Changing the Game for Traders.pptx
 
Harmonic pattern scanner mt5
Harmonic pattern scanner mt5Harmonic pattern scanner mt5
Harmonic pattern scanner mt5
 
Ags AIforTrading
Ags AIforTradingAgs AIforTrading
Ags AIforTrading
 
Features Enhance trades & maximize gains
Features Enhance trades & maximize gainsFeatures Enhance trades & maximize gains
Features Enhance trades & maximize gains
 
Winning trading logic – detecting supply and demand unbalance
Winning trading logic – detecting supply and demand unbalanceWinning trading logic – detecting supply and demand unbalance
Winning trading logic – detecting supply and demand unbalance
 
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdf
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdfAdding Peak Trough Analysis Tool in your Chart – MetaTrader.pdf
Adding Peak Trough Analysis Tool in your Chart – MetaTrader.pdf
 
Quantopian: Crowd-sourced Systematic Alpha
Quantopian: Crowd-sourced Systematic AlphaQuantopian: Crowd-sourced Systematic Alpha
Quantopian: Crowd-sourced Systematic Alpha
 
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017
Ashirth Barthur, Security Scientist, H2O, at MLconf Seattle 2017
 
Machine Learning and RoboCop Testing
Machine Learning and RoboCop TestingMachine Learning and RoboCop Testing
Machine Learning and RoboCop Testing
 
Atom 2018-share-v1
Atom 2018-share-v1Atom 2018-share-v1
Atom 2018-share-v1
 
Algorithmic Trading: an Overview
Algorithmic Trading: an Overview Algorithmic Trading: an Overview
Algorithmic Trading: an Overview
 
Algorithmic Trading
Algorithmic TradingAlgorithmic Trading
Algorithmic Trading
 
Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1
 
Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1Elliott wave trend basic tutorial 1
Elliott wave trend basic tutorial 1
 
Grid Forex Strategies
Grid Forex StrategiesGrid Forex Strategies
Grid Forex Strategies
 

More from James Urquhart

Flow Architectures: An Introduction
Flow Architectures: An IntroductionFlow Architectures: An Introduction
Flow Architectures: An IntroductionJames Urquhart
 
Deliver agile flow presentation (1)
Deliver agile   flow presentation (1)Deliver agile   flow presentation (1)
Deliver agile flow presentation (1)James Urquhart
 
Jurquhart o reilly_radar_preso_3
Jurquhart o reilly_radar_preso_3Jurquhart o reilly_radar_preso_3
Jurquhart o reilly_radar_preso_3James Urquhart
 
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...The Outcome Conversation: Aligning DevOps and Business Through Understanding ...
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...James Urquhart
 
Defrag2014 anomalies final
Defrag2014 anomalies finalDefrag2014 anomalies final
Defrag2014 anomalies finalJames Urquhart
 
Dev opsdayssv2014 devopsasrelationshipmanagement
Dev opsdayssv2014 devopsasrelationshipmanagementDev opsdayssv2014 devopsasrelationshipmanagement
Dev opsdayssv2014 devopsasrelationshipmanagementJames Urquhart
 
Cloud Computing for Business - The Road to IT-as-a-Service
Cloud Computing for Business - The Road to IT-as-a-ServiceCloud Computing for Business - The Road to IT-as-a-Service
Cloud Computing for Business - The Road to IT-as-a-ServiceJames Urquhart
 
Glue con2011 future_of_net_systems
Glue con2011 future_of_net_systemsGlue con2011 future_of_net_systems
Glue con2011 future_of_net_systemsJames Urquhart
 
Cloud and the Future of Networked Systems
Cloud and the Future of Networked SystemsCloud and the Future of Networked Systems
Cloud and the Future of Networked SystemsJames Urquhart
 
The New DevOps Designers: Cloud and The Big Rethink
The New DevOps Designers: Cloud and The Big RethinkThe New DevOps Designers: Cloud and The Big Rethink
The New DevOps Designers: Cloud and The Big RethinkJames Urquhart
 

More from James Urquhart (11)

Flow Architectures: An Introduction
Flow Architectures: An IntroductionFlow Architectures: An Introduction
Flow Architectures: An Introduction
 
Deliver agile flow presentation (1)
Deliver agile   flow presentation (1)Deliver agile   flow presentation (1)
Deliver agile flow presentation (1)
 
Jurquhart o reilly_radar_preso_3
Jurquhart o reilly_radar_preso_3Jurquhart o reilly_radar_preso_3
Jurquhart o reilly_radar_preso_3
 
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...The Outcome Conversation: Aligning DevOps and Business Through Understanding ...
The Outcome Conversation: Aligning DevOps and Business Through Understanding ...
 
Defrag2014 anomalies final
Defrag2014 anomalies finalDefrag2014 anomalies final
Defrag2014 anomalies final
 
Dev opsdayssv2014 devopsasrelationshipmanagement
Dev opsdayssv2014 devopsasrelationshipmanagementDev opsdayssv2014 devopsasrelationshipmanagement
Dev opsdayssv2014 devopsasrelationshipmanagement
 
Mining for gold 2.0
Mining for gold 2.0Mining for gold 2.0
Mining for gold 2.0
 
Cloud Computing for Business - The Road to IT-as-a-Service
Cloud Computing for Business - The Road to IT-as-a-ServiceCloud Computing for Business - The Road to IT-as-a-Service
Cloud Computing for Business - The Road to IT-as-a-Service
 
Glue con2011 future_of_net_systems
Glue con2011 future_of_net_systemsGlue con2011 future_of_net_systems
Glue con2011 future_of_net_systems
 
Cloud and the Future of Networked Systems
Cloud and the Future of Networked SystemsCloud and the Future of Networked Systems
Cloud and the Future of Networked Systems
 
The New DevOps Designers: Cloud and The Big Rethink
The New DevOps Designers: Cloud and The Big RethinkThe New DevOps Designers: Cloud and The Big Rethink
The New DevOps Designers: Cloud and The Big Rethink
 

Recently uploaded

UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGlokeshwarmaha
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...AustraliaChapterIIBA
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Winbusinessin
 
Project Brief & Information Architecture Report
Project Brief & Information Architecture ReportProject Brief & Information Architecture Report
Project Brief & Information Architecture Reportamberjiles31
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakEditores1
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)tazeenaila12
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toumarfarooquejamali32
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentationbaron83
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...Brian Solis
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...IMARC Group
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024Stephan Koning
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinAnton Skornyakov
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxJemalSeid25
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfHajeJanKamps
 
Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..dlewis191
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxWorkforce Group
 

Recently uploaded (20)

UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
 
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
IIBA® Melbourne - Navigating Business Analysis - Excellence for Career Growth...
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024
 
Project Brief & Information Architecture Report
Project Brief & Information Architecture ReportProject Brief & Information Architecture Report
Project Brief & Information Architecture Report
 
WAM Corporate Presentation Mar 25 2024.pdf
WAM Corporate Presentation Mar 25 2024.pdfWAM Corporate Presentation Mar 25 2024.pdf
WAM Corporate Presentation Mar 25 2024.pdf
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerak
 
Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024Borderless Access - Global B2B Panel book-unlock 2024
Borderless Access - Global B2B Panel book-unlock 2024
 
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
Harvard Business Review.pptx | Navigating Labor Unrest (March-April 2024)
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb to
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
MoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor PresentationMoneyBridge Pitch Deck - Investor Presentation
MoneyBridge Pitch Deck - Investor Presentation
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
 
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 Building Your Personal Brand on LinkedIn - Expert Planet-  2024 Building Your Personal Brand on LinkedIn - Expert Planet-  2024
Building Your Personal Brand on LinkedIn - Expert Planet- 2024
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
Slicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup BerlinSlicing Work on Business Agility Meetup Berlin
Slicing Work on Business Agility Meetup Berlin
 
Introduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptxIntroduction to The overview of GAAP LO 1-5.pptx
Introduction to The overview of GAAP LO 1-5.pptx
 
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdfPDT 89 - $1.4M - Seed - Plantee Innovations.pdf
PDT 89 - $1.4M - Seed - Plantee Innovations.pdf
 
Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..Team B Mind Map for Organizational Chg..
Team B Mind Map for Organizational Chg..
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
 

Don't let this happen to you! Cloud, complexity and drift

Editor's Notes

  1. Instead, let me take you back to May 6, 2010…
  2. A typical morning on the stock market. No major breaking news, and all boards trading normally. A slight drop in the indexes, but nothing special. But something went very wrong that afternoon.
  3. According to a joint report written later about that day by the Securities and Exchange Commission and the Commodity Futures Trading Commission, a single trading algorithm was used to mete out $4.1 billion dollars in trades, using an algorithm that metered out individual trades over time, attempting to represent no more than 10% of trading volume at any given time.To achieve this, the algorithm adjusted each trade’s volume based on overall market volume in the previous minute.
  4. Unfortunately, for one reason or another…a simple bug, perhaps, or human error…trades that were meant to be metered out over days or weeks were actually executed within 20 minutes.As might be expected, this resulted in some pretty big trade executions in a very short period of time, especially for the relatively small electronic exchange on which they were executed.The market responded, and other automatic trading algorithms sensed a “sell” signal, and started executing sell trades in response. This increased market volume.The original sell algorithm then responded to the increased trade volume, and increased it’s own trading volume.
  5. The result was an about 4% drop in the Dow in about 20 minutes, a total of 8.1% down from the opening bell. To put that into context, CNN noted that it was the biggest intraday point drop in Dow Jones history at that time.
  6. So, to fix this, we need the root cause, right? I mean, it had to be that original trading company’s bug or bad logic or whatever. Case closed.
  7. Well, it was that errant trading algorithm, right? I mean it initiated the trades, and feedback from that market was then used to initiate further trades.
  8. Then, for whatever reason, other trading algorithms quickly saw this larger than normal activity as a sell sign…
  9. …so they in turn initiated trades on both the original exchange, and (in a few cases) on other exchange mechanisms.
  10. Those trades were either frequent enough or large enough to get the attention of yet more trading algorithms.
  11. A few of those algorithms probably triggered increased trading on the original exchange, further exacerbating the problem.Now, this is an extremely simplified view of the changing state of the exchange system that day, and one that appears much more serial than events really unfolded.
  12. But at this point, how can we say that first algorithm was the “problem” to be “fixed”.In fact, the truth is that it’s the way the other algorithms reacted to the initial trades that made those trades a problem. In theory, any one of a number of large trade sources could have triggered the same series of events. Or a similar series of events. Or, perhaps, an even more devastating series of events.Another important truth is that many, many decisions were made in parallel, often affecting large numbers of trades made by entirely unrelated parties. And there were a huge number of parties. The actual trading graph probably looked more like…
  13. …this. A huge web of players interacting over a variety of paths via a broad range of rules and protocols.
  14. In systems like this, small actions can trigger huge consequences. And most of the time, it’s not the initial trigger that is the problem. In this case, the root cause is interesting, but it’s not the problem.
  15. The Flash Crash post mortem itself noted that “May 6 was an important reminder of the interconnectedness of our derivatives and securities markets”.
  16. “Interconnectedness”. I love that word. Isn’t that what we are all here to talk about this week? Isn’t that what the world of computing is working towards with such fervor and focus?
  17. I mean, the Internet is all about interconnectedness…from the early days of The World Wide Web through human social networks to—increasingly—system to system, device-to-device, thing-to-thing connectedness.
  18. APIs are also a good example of what drives interconnectedness in computing. Simple interfaces to powerful services are driving intense acceleration in the linking of software systems.
  19. And cloud computing is an interesting use case for interconnectedness. Cloud is bringing together computing systems at a scale never before seen. WE are defining perhaps the most interconnected knowledge system in the history of humanity. The cloud is starting to look like…
  20. …this. A huge web of players interacting over a variety of paths via a broad range of rules and protocols.Oh, my. Computing is increasingly looking like the stock market. (How long until we see “certified capacity planners” cold calling us to “manage our portfolios”. Perhaps it’s already begun.)OK, so what’s going on here?
  21. Both the cloud and the stock market are examples of a phenomenon that has received a tremendous amount of scientific interest in the past few decades: complex adaptive systems.
  22. Wait. What? That word makes most people cringe like a Dell PR rep…
  23. …when they hear the name Mads Christiansen.Don’t worry. In this context, it’s not nearly as obscene.
  24. So, what the hell arecomplex adaptive systems?
  25. You experience complex adaptive systems every day. In ecosystems…
  26. …in societies,…
  27. …and in economies. These are just a few examples.Complex adaptive systems are often big, mysterious systems that just seem to work (except when they don’t …more on that later).However, for as large and complicated as they appear, complex adaptive systems are made up of just a few basic elements.
  28. The first is a large number of individual agents…
  29. Agents are defined as entities that operate individually, responding to feedback from the world around them and taking action as their own internal state and rules dictate.So, for example, in computing, an example might be a component that responds to input and takes action as it’s own program dictates.There can be a variety of agent types in a system, for example in computing, some may be mobile clients, services, management agents, servers, disk filers, and so on.When systems are adaptive, the agents can also learn from the feedback they receive, adjusting rules in an attempt to achieve whatever results are optimal for the agent.So, in ecology, organisms adapt to survive and reproduce. In markets, business adapt to profit and grow.The critical thing here is that these agents are not directly controlled by any outside force. Each agent makes it’s own “decisions” on action to take in response to input.
  30. The second element is the way these agents interact with each other.
  31. Relationships between agents change over time, sometimes quite frequently, getting created or destroyed as the agents decide which relationships benefit them the most, or—occasionally—when the system itself forces a relationship to change.If you were to graph out the relationships between agents over the system as a whole, however, you’d probably discover some patterns emerging, depending on the nature of the system.Some systems organize into complex hub-and-spoke networks, like the one above.Others organize into fields in which there are few central players, but a large number of connections between individual agents on an ad hoc basis. There are hierarchical graphs, and more.Regardless, the agents find each other, and begin to interact in ways that begin to create a cohesive system of agents.
  32. Finally, the agents work off of some (usually)simple set of rules.
  33. Now, these rules could be defined differently for different types of agents, and even individual agents may see slight variances from its peers.Nonetheless, these rules are typically defined to benefit the class of agent or the individual agent.Some rules are about how the agent should send signals to other agents. Other rules are about how the agent should evaluate and adjust its behavior. Still others are about how the agent itself should change in response to its environment. I’m simplifying here, but it’s critical to understand that there are no central controllers in complex adaptive systems. It’s all about how the agents interact.
  34. All of this results in something spectacular.
  35. With independent agents, dynamic interaction and rules for that interaction, you get an agent that demonstrates emergent behavior.
  36. An excellent example of this emergent behavior is a flock of birds. There are no leaders in most bird flocks. Rather each bird is responding to a simple set of rules ingrained in it’s DNA. Those rules create a beautiful, fluid movement that almost behaves like a single entity. A system is born from individual agents, dynamic interaction and simple rules.
  37. Another interesting thing about these systems is that they can typically be modeled. Maybe not perfectly, but certainly in ways that are helpful in evaluating the system itself.
  38. Going back to our bird flock example, there is a phenomenal example model that was introduced by computer scientist Craig Reynolds in 1986.Three elements are managed for any given bird: separation from other birds (not too close), cohesion (not too far), and alignment (try to head in the same direction as neighboring birds).The resulting model is amazing to watch in action. A flocking behavior that seems so “real”, that you can’t help but believe you’re seeing a natural flocking behavior.The financial industry, by the way, is famous for modeling large parts of the stock markets and the overall economy itself. Those models are actually getting pretty good, but they won’t create any magic way to predict future markets. Why?
  39. Because the last trait of complex adaptive systems is that there sheer scale and decentralized nature makes it almost impossible to predict the future state of the system.
  40. In late 2010 and early 2011, hundreds of birds were found dead in states ranging from California to Georgia. The cause? No one knows. In some cases, people witnessed birds divebombing semi-trucks. One witness said “it looked like they were committing suicide.” Could anyone have predicted this before it happened?
  41. The Deepwater Horizon disaster is a classic example of failure caused by systems flaws that were unseen or ignored until the disaster happened. The decisions about safety and technologies leading up to the explosion were made by people who believed those decisions were smart and safe. Hell, they were running with those technologies for years in tests and the early life of the platform without problems. What reason did they have to predict a failure—except in retrospect.
  42. The flash crash, of course, is a great example of the unpredictability of systems automation and computing when a large number of individual programs interact over a network against the same data. No matter how well you tighten each individual algorithm, the system as a whole can have hidden traps that are only discovered in extreme conditions.
  43. All of this means that, if you are responsible for a critical component of your companies IT infrastructure running in this complex systems environment, you look a little like this guy.
  44. So…what can you do?
  45. Well, the first thing you can do is embrace complexity. The typical response to this is…
  46. “Hell no! Complexity is evil!”
  47. So, how about if we get a little more specific. Embrace complex systems. Specifically embrace complex adaptive systems, and all that we have learned about how they work and how we can take advantage of their properties.
  48. If you want a keyword to search for solutions that target applying complex systems science to everything from organizations to processes to computing, the term commonly used is “Systems Thinking”.
  49. At this point, it would probably be good for me to get out of theory a little bit, and start talking about practical steps you can take to embrace complexity…er, complex systems.
  50. First, please, please, please, take some time to learn about complex adaptive systems and systems thinking.
  51. Here are three excellent works to start with. “Complexity”, on the left, is an amazing telling of how the Sante Fe Institute, an academic research institution that gathers the best and the brightest in a variety of disciplines to explore the effect of complex adaptive systems on their respective fields, giving a phenomenal overview of the science along the way. “Thinking in Systems” is one of the most respected introductions to Systems Thinking, and Drift into Failure makes clear the dangers and pitfalls that await us all when working with complex systems.
  52. Second, pay attention to one of the most important tradeoffs in complex systems, and choose resilience.
  53. A seminal paper about the pathology of direct management of natural ecosystems proved pretty conclusively that targeting a stable high yield will often result in reduced resilience.
  54. This tradeoff between stability and resilience is critical to understand. If you work towards stability—an environment where any form of change is discouraged in favor of completely predictable outcomes, you’ll find your applications and services actually becoming more susceptible to conditions outside of anticipated norms. When that happens, “stable” systems tend to collapse entirely. Systems build for resilience, on the other hand, have failures all the time, but are build so that those failures have a minimum impact on the system as a whole. So the system remains resilient.
  55. One of the best examples of design for resilience succeeding is the Amazon.com home page. Each of those features you see up there: featured offers, highlighted ads, personalized “what’s hot” sections, are all separate components working within a larger system. If one component fails, either other components fill the need, or the feature is just not displayed. The result is obvious…when was the last time you went to Amazon.com and it just wasn’t there?
  56. Third, understand that, to build agents that thrive in complex systems, you need to focus much of your engineering not on the agent itself, but on how that agent relates to the world around it.
  57. Remember the flash crash model I talked about earlier? What were the things that could be changed that would avoid the likelihood of a similar incident a few months or years later? Not the individual algorithms (hell, the financial services companies would never share those, anyways). Nope…
  58. …it’s the relationships between those components (including many components not included in this model). The first thing the exchanges looked at (as well as the SEC and the Commodity Futures Trading Commission) was what mechanisms they could put in place to capture the “system” going haywire and stop a crisis before it starts.
  59. In computing, believe it or not, there is already an excellent pattern that has proven itself in large scale computing environments that does exactly that. From a book called “Release IT!” which is another read I highly recommend, the Circuit Breaker pattern is all about intercepting traffic at an API or on a network, applying rules to that traffic and triggering certain actions, like forcing a failure of the API call, when it sees something it doesn’t like.
  60. I wish I had more time to tell you about Circuit Breakers, but luckily one of the pioneers of its use at high scale has shared most of what they did in implementing them. I highly recommend checking out this link and the rest of the Netflix techblog if you want to learn more.
  61. Finally, one of the hardest things you’ll have to do as a developer operating in a complex adaptive system is to stay disciplined and aware enough of the system to avoid a drift into critical failures.
  62. Drift is everywhere in complex systems. It is a gradual shift in the structure of a system that actually makes failure MORE likely, even if the goal was to make it less likely. The Deepwater Horizon explosion and subsequent oil spill is an example that Sydney Dekker calls out in his book, that I mentioned earlier. The BT employees interviewed after the accident often mentioned they were uncomfortable with specific decisions, given other factors they knew about, but felt it wasn’t worth the political cost to call them out, and besides—those decisions were often in place for months or years (on a variety of platforms) without negative consequence, so perhaps it wasn’t so risky after all…
  63. Here’s how you avoid disaster in your production systems: beat the hell out of them…
  64. The concept of “Chaos Monkeys” was made famous in part through the practices of companies like Amazon and Netflix. They actually continuously run test systems that stress the system from a usage perspective, perhaps degrade the performance of one component or another, or even outright kill components just to see what happens. The result is that developers KNOW there will be challenges to the components they are building, and they begin to architect them to be resilient…not stable.
  65. Another great idea to keep in mind to avoid drift: remember that the component you are building is NOT the system, but an agent in a larger system.
  66. This is really quite different from the “build the most stable application you can” mentality of the siloed world. In an app-centric world, the focus is on the component—the application. The result is all about stabilizing the design with assumptions that the world around the application can be controlled. In cloud, that won’t work. You HAVE to take time to instrument as much of the world around your app or service or whatever as you can. Use that data as feedback, both in component design and operation, as well as in tweaking the system as much as you can to help your stuff thrive.Not survive. Thrive.
  67. One last thing. The hardest thing to remember when working with complex systems is that everything that touches the system is (generally) a part of that system. That means you are a part of the system, and can only have—at best—a limited understanding of how the system works as a whole. It’s as if you were standing in a vast forest of tall trees. Quick. How big is the forest?Embrace complexity—er, complex adaptive systems. Don’t let this happen to you…
  68. Thank you.