SlideShare a Scribd company logo
1 of 43
Download to read offline
Rollback
The Impossible Dream

    by James Turnbull

     jamtur01 @ github
       kartar @ twitter
 jamesturnbull on freenode
  james @ puppetlabs.com
About Me
VP Technical Operations at Puppet Labs

             Puppet guy

              Ruby guy

             Talks funny
A show of hands
Who thinks they
know what rollback
       is?
Last set of hands
YMMV
Definitions
Traditional
Modern
Fact or Fiction?
Accept certain
 constraints
Constraint #1
Apply sufficient
    capital
Constraint #2
 Idempotent
Constraint #3
Cascade-less failure
Constraint #4
 Resources
A Philosophical
  Digression
If I know where I am
I don’t know how I got there
   If I know how I got there
   I don’t know where I am
Very few “systems”
are truly deterministic
A Mathematical
  Digression
On system rollback and totalised fields
     An algebraic approach to system change
          Mark Burgess and Alva Couch
                 20th June 2011

http://cfengine.com/markburgess/papers/totalfield.p
                        df
So what’s wrong with
     rollback?
Risk
Learning from
  mistakes
Complex systems are
   … complex
Human error
What is the problem
rollback is trying to
       solve?
What is the problem
 YOU are trying to
      solve?
So how can we
mitigate Rollback
 shortcomings?
Preventative
  Design
Rollback is (often) an
architecture problem
Increase Resilience
Operational
Intelligence
A little bit of DevOps in
       every byte…
Small, iterative
  changes
Accept that failure
    happens
“We can’t test that?
 Okay we can roll it
 back if it breaks…”
Assumption is the
mother of all fuckups*
“But the system can’t
           be
{run|upgraded|deploy
ed} like that because…”
Conclusions
Rollback is possible
 but not probable
If you have to have
  “rollback” accept
     constraints
You can mitigate the
     need for it
Thank you!


Questions/Insults?
     jamtur01 @ github
       kartar @ twitter
 jamesturnbull on freenode
  james @ puppetlabs.com

More Related Content

Viewers also liked

Why Monitoring Sucks and what Configuration Management can do about it
Why Monitoring Sucks and what Configuration Management can do about itWhy Monitoring Sucks and what Configuration Management can do about it
Why Monitoring Sucks and what Configuration Management can do about itJames Turnbull
 
State of the Puppet Community - PuppetConf 2012
State of the Puppet Community - PuppetConf 2012State of the Puppet Community - PuppetConf 2012
State of the Puppet Community - PuppetConf 2012James Turnbull
 
Once upon a time: Why operations mythology matters
Once upon a time: Why operations mythology mattersOnce upon a time: Why operations mythology matters
Once upon a time: Why operations mythology mattersJames Turnbull
 
Monitoring As A Service - Modernity and Self-Service CraftConf 2016
Monitoring As A Service - Modernity and Self-Service CraftConf 2016Monitoring As A Service - Modernity and Self-Service CraftConf 2016
Monitoring As A Service - Modernity and Self-Service CraftConf 2016James Turnbull
 
Developing Good Operations Tools
Developing Good Operations ToolsDeveloping Good Operations Tools
Developing Good Operations ToolsJames Turnbull
 
Monitoring As a Service
Monitoring As a ServiceMonitoring As a Service
Monitoring As a ServiceJames Turnbull
 
LogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesomeLogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesomeJames Turnbull
 
Monitoring As A Service - Monitorama 2015
Monitoring As A Service - Monitorama 2015Monitoring As A Service - Monitorama 2015
Monitoring As A Service - Monitorama 2015James Turnbull
 
A Gentle Introduction To Docker And All Things Containers
A Gentle Introduction To Docker And All Things ContainersA Gentle Introduction To Docker And All Things Containers
A Gentle Introduction To Docker And All Things ContainersJérôme Petazzoni
 
Docker introduction
Docker introductionDocker introduction
Docker introductiondotCloud
 
Monitoring as a service - Velocity NYC 2015
Monitoring as a service - Velocity NYC 2015Monitoring as a service - Velocity NYC 2015
Monitoring as a service - Velocity NYC 2015James Turnbull
 

Viewers also liked (11)

Why Monitoring Sucks and what Configuration Management can do about it
Why Monitoring Sucks and what Configuration Management can do about itWhy Monitoring Sucks and what Configuration Management can do about it
Why Monitoring Sucks and what Configuration Management can do about it
 
State of the Puppet Community - PuppetConf 2012
State of the Puppet Community - PuppetConf 2012State of the Puppet Community - PuppetConf 2012
State of the Puppet Community - PuppetConf 2012
 
Once upon a time: Why operations mythology matters
Once upon a time: Why operations mythology mattersOnce upon a time: Why operations mythology matters
Once upon a time: Why operations mythology matters
 
Monitoring As A Service - Modernity and Self-Service CraftConf 2016
Monitoring As A Service - Modernity and Self-Service CraftConf 2016Monitoring As A Service - Modernity and Self-Service CraftConf 2016
Monitoring As A Service - Modernity and Self-Service CraftConf 2016
 
Developing Good Operations Tools
Developing Good Operations ToolsDeveloping Good Operations Tools
Developing Good Operations Tools
 
Monitoring As a Service
Monitoring As a ServiceMonitoring As a Service
Monitoring As a Service
 
LogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesomeLogStash - Yes, logging can be awesome
LogStash - Yes, logging can be awesome
 
Monitoring As A Service - Monitorama 2015
Monitoring As A Service - Monitorama 2015Monitoring As A Service - Monitorama 2015
Monitoring As A Service - Monitorama 2015
 
A Gentle Introduction To Docker And All Things Containers
A Gentle Introduction To Docker And All Things ContainersA Gentle Introduction To Docker And All Things Containers
A Gentle Introduction To Docker And All Things Containers
 
Docker introduction
Docker introductionDocker introduction
Docker introduction
 
Monitoring as a service - Velocity NYC 2015
Monitoring as a service - Velocity NYC 2015Monitoring as a service - Velocity NYC 2015
Monitoring as a service - Velocity NYC 2015
 

Similar to Rollback: The Impossible Dream

Architecting a Post Mortem - Velocity 2018 San Jose Tutorial
Architecting a Post Mortem - Velocity 2018 San Jose TutorialArchitecting a Post Mortem - Velocity 2018 San Jose Tutorial
Architecting a Post Mortem - Velocity 2018 San Jose TutorialWill Gallego
 
Real-world Reinforcement Learning
Real-world Reinforcement LearningReal-world Reinforcement Learning
Real-world Reinforcement LearningMax Pagels
 
Real-world Reinforcement Learning
Real-world Reinforcement LearningReal-world Reinforcement Learning
Real-world Reinforcement LearningMax Pagels
 
Reaction Paper On Mind Hacking
Reaction Paper On Mind HackingReaction Paper On Mind Hacking
Reaction Paper On Mind HackingJessica Hill
 
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...John Mathon
 
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech GacaUNICORNS IN TECH
 
Monte Carlo on a Blackpool Budget
Monte Carlo on a Blackpool BudgetMonte Carlo on a Blackpool Budget
Monte Carlo on a Blackpool BudgetPaul Briërley
 
Gamification of Chaos Testing
Gamification of Chaos TestingGamification of Chaos Testing
Gamification of Chaos TestingBram Vogelaar
 
Chaos Engineering, When should you release the monkeys?
Chaos Engineering, When should you release the monkeys?Chaos Engineering, When should you release the monkeys?
Chaos Engineering, When should you release the monkeys?Thoughtworks
 
The Art Of Performance Tuning - with presenter notes!
The Art Of Performance Tuning - with presenter notes!The Art Of Performance Tuning - with presenter notes!
The Art Of Performance Tuning - with presenter notes!Jonathan Ross
 
Continuous Automated Testing - Cast conference workshop august 2014
Continuous Automated Testing - Cast conference workshop august 2014Continuous Automated Testing - Cast conference workshop august 2014
Continuous Automated Testing - Cast conference workshop august 2014Noah Sussman
 
Fabien Girardin PICNIC07
Fabien Girardin PICNIC07Fabien Girardin PICNIC07
Fabien Girardin PICNIC07crossmediaweek
 
Claudio Perrone - Outstanding Presentations
Claudio Perrone - Outstanding PresentationsClaudio Perrone - Outstanding Presentations
Claudio Perrone - Outstanding PresentationsBizcamp South East
 
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...DataScienceConferenc1
 
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020Dave Aronson
 
Guerrilla portfolio management
Guerrilla portfolio managementGuerrilla portfolio management
Guerrilla portfolio managementAlberto Brandolini
 
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...Fwdays
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersJonas Bonér
 
The Oracle Advisors from a Different Perspective
The Oracle Advisors from a Different PerspectiveThe Oracle Advisors from a Different Perspective
The Oracle Advisors from a Different PerspectiveKaren Morton
 
Kill All Mutants KCDC 2021
Kill All Mutants KCDC 2021Kill All Mutants KCDC 2021
Kill All Mutants KCDC 2021Dave Aronson
 

Similar to Rollback: The Impossible Dream (20)

Architecting a Post Mortem - Velocity 2018 San Jose Tutorial
Architecting a Post Mortem - Velocity 2018 San Jose TutorialArchitecting a Post Mortem - Velocity 2018 San Jose Tutorial
Architecting a Post Mortem - Velocity 2018 San Jose Tutorial
 
Real-world Reinforcement Learning
Real-world Reinforcement LearningReal-world Reinforcement Learning
Real-world Reinforcement Learning
 
Real-world Reinforcement Learning
Real-world Reinforcement LearningReal-world Reinforcement Learning
Real-world Reinforcement Learning
 
Reaction Paper On Mind Hacking
Reaction Paper On Mind HackingReaction Paper On Mind Hacking
Reaction Paper On Mind Hacking
 
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...
Artificial Intelligence is back, Deep Learning Networks and Quantum possibili...
 
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca
8 PEOPLE, 4 SERVICES, 40 COUNTRIES - Wojciech Gaca
 
Monte Carlo on a Blackpool Budget
Monte Carlo on a Blackpool BudgetMonte Carlo on a Blackpool Budget
Monte Carlo on a Blackpool Budget
 
Gamification of Chaos Testing
Gamification of Chaos TestingGamification of Chaos Testing
Gamification of Chaos Testing
 
Chaos Engineering, When should you release the monkeys?
Chaos Engineering, When should you release the monkeys?Chaos Engineering, When should you release the monkeys?
Chaos Engineering, When should you release the monkeys?
 
The Art Of Performance Tuning - with presenter notes!
The Art Of Performance Tuning - with presenter notes!The Art Of Performance Tuning - with presenter notes!
The Art Of Performance Tuning - with presenter notes!
 
Continuous Automated Testing - Cast conference workshop august 2014
Continuous Automated Testing - Cast conference workshop august 2014Continuous Automated Testing - Cast conference workshop august 2014
Continuous Automated Testing - Cast conference workshop august 2014
 
Fabien Girardin PICNIC07
Fabien Girardin PICNIC07Fabien Girardin PICNIC07
Fabien Girardin PICNIC07
 
Claudio Perrone - Outstanding Presentations
Claudio Perrone - Outstanding PresentationsClaudio Perrone - Outstanding Presentations
Claudio Perrone - Outstanding Presentations
 
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
[DSC Europe 23] Danijela Horak - The Innovator’s Dilemma: to Build or Not to ...
 
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020
"Kill All Mutants! (Intro to Mutation Testing)" Slides from NDC Sydney 2020
 
Guerrilla portfolio management
Guerrilla portfolio managementGuerrilla portfolio management
Guerrilla portfolio management
 
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...
Jeff Atwood, Michael Krakovskiy "The role of catastrophic failure in software...
 
Without Resilience, Nothing Else Matters
Without Resilience, Nothing Else MattersWithout Resilience, Nothing Else Matters
Without Resilience, Nothing Else Matters
 
The Oracle Advisors from a Different Perspective
The Oracle Advisors from a Different PerspectiveThe Oracle Advisors from a Different Perspective
The Oracle Advisors from a Different Perspective
 
Kill All Mutants KCDC 2021
Kill All Mutants KCDC 2021Kill All Mutants KCDC 2021
Kill All Mutants KCDC 2021
 

Recently uploaded

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Rollback: The Impossible Dream

Editor's Notes

  1. Dev / Ops / QA / Security? / Others?
  2. ??
  3. Does anyone rely on rollback?
  4. This is very much opinion based on experience. Everyone’s shop is different – everyone has different constraints and requirements.A trading house differs from a twitter analytics company differs from a hospital from a .gov/Fed.Distance the discussion from ”the sometimes emotional standpoints that bind system administrators to the notion of rollback: desperately wanting does not make it possible”But every shop has technical heritage and technical debtEstablished institutional memory/remembered painApproach with an open mind and don’t make assumptionsWelcome new ideas and evaluate old constructsYou don’t have to agree / you can think I am a clueless idiot – as long as you do so based on clear, established data not “we’re a different special snowflake” because you’re fucking not.
  5. Changed my views a little since writing the abstract.
  6. Trad/Modern – arbitrary labelsDatabase rollback, transactional rollbackIn single-threaded and parallel software applications, many authors have developed a ‘journaling’ approach to reversibility and rollback (see foregoing references on checkpointing). A stackof state-history can be kept to arbitrary accuracy (and at proportional cost), provided there is sufficient memory to document changes.
  7. Service rollback. Many interconnecting components.Interconnectness between application(s) and infrastructure changesRelease management, Checkpointing, Snapshots and version control In more general ‘open’ (or incompletely specified) systems the cost of maintaining history increases without bound as system complexity increases.
  8. Rollback isn’t a myth – for certain definitions in certain circumstances it MAY be possible to do something that resembles a rollback.
  9. Roll-back recovery requires that the operations between the checkpoint and the detected erroneous state can be made idempotent.
  10. Apply enough money and set enough constraints and you can have something like rollback.Duplicate infrastructure / scale
  11. Roll-back recovery requires that the operations between the checkpoint and the detected erroneous state can be made idempotent.
  12. A cascading rollback occurs in database systems when a transaction (T1) causes a failure and a rollback must be performed. Other transactions dependent on T1's actions must also be rollbacked due to T1's failure, thus causing a cascading effect. That is, one transaction's failure causes many to fail.Practical database recovery techniques guarantee cascadeless rollback, therefore a cascading rollback is not a desirable result.
  13. You must have sufficient memory/storage/resources to maintain sufficient history to rollback to a specified point
  14. Story about University and “You are here” signs. Promised Heisenberg – uncertainty principle lower bound on the precision on which certain pairs of properties of particles can be measured (location / speed). The closer you measure one the harder it is to measure the other. Observer principle – observing things actually resulting in making it hard to measure them.
  15. Story about University and “You are here” signs. Promised Heisenberg – uncertainty principle lower bound on the precision on which certain pairs of properties of particles can be measured (location / speed). The closer you measure one the harder it is to measure the other. Observer principle – observing things actually resulting in making it hard to measure them
  16. A deterministic system is one in which no randomness in the development of future states of the system. Lessons learnt about Complex systems and systems thinking.
  17. I have a Liberal Arts degree and got someone sciency and smart to explain the hard bits to me.
  18. Risk – false sense of security
  19. Unless you are committed to testing 'rollback' on a regular basis,maybe even every deploy, you inevitably end up in a situation where atthe worst possible moment you are going to be depending on a processthat is rarely done.We backup but we never restore.We have UPS/Genneratot but we’ve never tested itWe’ve got DRP but it’s too difficult/dangerous to execute it.
  20. No matter how much you believe things can be tracked there is always something that either can’t be tracked, can’t be predicted or is simply unknown.Deterministic reference.
  21. K.I.S.S – rollback changes are usually made after a production changes fails, when the team is at a low, often tired, often frustrated, often angry.
  22. Return the system to a known good state removing any erroneous transactions from the systems
  23. Return the system to a known good state removing/correcting any erroneous transactions from the systems AND return the system to working order as fast as possible.Are these different? Contradictory?
  24. Dev / Ops / QA
  25. Dev / Ops / QA
  26. http://www.slideshare.net/mmalone/architecture-at-simplegeo-staying-agile-at-scaleIf your system is hard to deploy or you can’t upgrade without org risk then that’s an architectural problem NOT an operational one
  27. http://www.slideshare.net/mmalone/architecture-at-simplegeo-staying-agile-at-scale
  28. Disrupt
  29. Continuous deployment on end of spectrum – other end is more small change rather than big bang change.If it hurts do it more until it stops hurting
  30. Accept failure, learn from it, move forward not backwards, you are going to have to deploy anything you roll back now again sometime in the future.
  31. Having rollback is not an excuse not to SUFFICIENTLY test
  32. Under Siege 2.Don’t assume the past dictates the futureLess NIH and religion – more science and data
  33. Poet John Lydgate – ably stolen by Abraham Lincoln“You can please some of the people all of the time, you can please all of the people some of the time, but you can’t please all of the people all of the time”.
  34. Or worth the effort.
  35. Don’t lie to yourself.from ”the sometimes emotional standpoints that bind system administrators to the notion of rollback: desperately wanting does not make it possible”Thank you.