Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous improvement devops day mountain view 2012


Published on

The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product's life cycle. When there is no QA phase, can there still be QA?

Indeed, as the speed of product development changes, risk must be assessed differently. A monthly release cycle provides time to assure a product is reasonably free of defects. In a rapid release cycle, such assurance is impossible. Quality instead comes to mean assuring that the product is capable of recovering from defects.

I will discuss how software risk mitigation practice changes as the speed of development increases. And I will explore the idea that recovering from failure is a far more pragmatic goal than preventing failure in the first place.

Published in: Technology
  • Finally found a service provider which actually supplies an essay with an engaging introduction leading to the main body of the exposition Here is the site ⇒⇒⇒ ⇐⇐⇐
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating direct: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here

Continuous improvement devops day mountain view 2012

  1. 1. ContinuousImprovement How rapid release cycles alter QA and testing. Noah SussmanDevopsDay Mountain View, 2012 @noahsussman #devopsdays
  2. 2. The Canonical Agile Release CycleSprints of two weeks or more in length.Start deployment process at the end of the sprint.QA is part of the deployment process.QA must be complete before new code goes live.
  3. 3. The Continuous Release CycleMinimum viable feature set.Deployment is decoupled from release.Real-time data on how releases impact revenue.Constant tweaks to live features.
  4. 4. Releasing a featureis decoupled fromdeploying code. David E. Smith
  5. 5. An airport withoutan air traffic controller. —Chad Dickerson
  6. 6. This Part Really IsDifferent From AgileLarge features are deployed piecemeal over time.Every feature is part of an A/B campaign.Dark and Admin-Only launches.Wire-Offs and Config Flags.There is no “Done Done.”
  7. 7. Observed BehaviorOf Complex SystemsEmergent behaviors require unplanned responses.Improvements, too are discovered not designed.Users of the system have complex expectations.Such systems are never “complete.”
  8. 8. QA Happens When?First of all, what is “Quality Assurance?”Authoritatively assuring that there are no defects?That’s impossible.
  9. 9. Testing is everyone’s job.
  10. 10. Myths About Bug DetectionThere are a finite number of bugs.There are a finite number of detectable bugs.All severity one bugs can be found before releaseSoftware is built to specifications.At some point, software is finished.
  11. 11. The Biggest MythBugs have complex, unpredictable causes.In fact, most errors in software are the results ofincorrect assumptions made by programmers.
  12. 12. Many SmallAnomalies CombinedThe “Swiss Cheese” model of risk presents uswith a strong case for prioritizing the elimination ofsmall errors rather than focusing on the mitigationof large catastrophic failures.Unit testing is great at eliminating small errors.
  13. 13. The whole time I’mprogramming, I’mconstantly checkingmy assumptions. —Rasmus Lerdorf
  14. 14. Resilience, Not “Quality”Readable code.Reasonable test coverage.Sane architecture.Good debugging tools.An engineering culture that values refactoring.These are measureable goals.
  15. 15. Manual TestingIt doesn’t always look like you think it looks.Real-Time Monitoring is the new face of testing.
  16. 16. Anomaly detection is hard.
  17. 17. Watching The GraphsEtsy collects well over a quarter million metrics.Deciding which ones matter is a human problem.Everyone watches some subset of the graphs.Human vision excels at anomaly detection.
  18. 18. QA Happens When??Exploratory testing can be performed any time.Rigorous, scientific approach.Focus on customer satisfaction rather than a spec.Equally useful before or after a release.
  19. 19. Just Quality“Assurance” is a terrible word. Let’s discard it.Quality exists, we just can’t assure or prove it.
  20. 20. There Is No Such Thing As A Formal Proof Of Quality.Yet most of us would agree it exists.I propose that “customer experience” is a betterterm-of-art-than “quality.”Though there’s no formal proof for that either.
  21. 21. Exploratory TestingAddresses areas that Developer Testing can’t.Developer Testing validates assumptions.The Tester’s job is to invalidate assumptions.
  22. 22. Technology InformsCustomer ExperienceExploratory Testing requires an understanding ofthe ways in which a whole system is intended toserve a community of users.The problem space has as much to do withtechnology as it does with product requirements.
  23. 23. Most bugs, most of thetime, are easily nailed giveneven an incomplete butsuggestive characterizationof their error conditions atsource-code level. —Eric S. Raymond
  24. 24. Source, diffs, logs.If your QA Analysts don’t look at these, teach them.
  25. 25. Customer SupportYour customer support operators spend more timetalking to your users than anyone else.Customer Support interface with users asindividuals rather than as aggregate data.
  26. 26. Keep the feedback loop short.
  27. 27. Manage Your Culture.
  28. 28. Effeciency To Thoroughness Trade-OffRapid release cycles have different risks thanslower release cycles.But nothing about risk itself has changed.
  29. 29. Test EverywhereForeseeable errors can be worked out in dev.Unforeseeable errors must be worked out in prod.
  30. 30. Fail ForwardLet go of the idea of “last stable release.”Software exists in context.Networks, services and people are always in flux.
  31. 31. Forget About Satisfying The RequirementsWatch your graphs.Listen to your customers.Adhere to your protocols.Improve your product.
  32. 32. Questions?
  33. 33. Further Reading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspaw“What Is Exploratory Testing?,” James Bach“How Many Eyeballs Tame Complexity,” ESR“The Timeless Way of Building,” Christopher Alexander