The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product's life cycle. When there is no QA phase, can there still be QA?
Indeed, as the speed of product development changes, risk must be assessed differently. A monthly release cycle provides time to assure a product is reasonably free of defects. In a rapid release cycle, such assurance is impossible. Quality instead comes to mean assuring that the product is capable of recovering from defects.
I will discuss how software risk mitigation practice changes as the speed of development increases. And I will explore the idea that recovering from failure is a far more pragmatic goal than preventing failure in the first place.
Continuous improvement devops day mountain view 2012
ContinuousImprovement How rapid release cycles alter QA and testing. Noah SussmanDevopsDay Mountain View, 2012 @noahsussman #devopsdays
The Canonical Agile Release CycleSprints of two weeks or more in length.Start deployment process at the end of the sprint.QA is part of the deployment process.QA must be complete before new code goes live.
The Continuous Release CycleMinimum viable feature set.Deployment is decoupled from release.Real-time data on how releases impact revenue.Constant tweaks to live features.
Releasing a featureis decoupled fromdeploying code. David E. Smith http://www.ﬂickr.com/photos/david_e_smith/3566697122
An airport withoutan air trafﬁc controller. —Chad Dickerson
This Part Really IsDifferent From AgileLarge features are deployed piecemeal over time.Every feature is part of an A/B campaign.Dark and Admin-Only launches.Wire-Offs and Conﬁg Flags.There is no “Done Done.”
Observed BehaviorOf Complex SystemsEmergent behaviors require unplanned responses.Improvements, too are discovered not designed.Users of the system have complex expectations.Such systems are never “complete.”
QA Happens When?First of all, what is “Quality Assurance?”Authoritatively assuring that there are no defects?That’s impossible.
Myths About Bug DetectionThere are a ﬁnite number of bugs.There are a ﬁnite number of detectable bugs.All severity one bugs can be found before releaseSoftware is built to speciﬁcations.At some point, software is ﬁnished.
The Biggest MythBugs have complex, unpredictable causes.In fact, most errors in software are the results ofincorrect assumptions made by programmers.
Many SmallAnomalies CombinedThe “Swiss Cheese” model of risk presents uswith a strong case for prioritizing the elimination ofsmall errors rather than focusing on the mitigationof large catastrophic failures.Unit testing is great at eliminating small errors.
The whole time I’mprogramming, I’mconstantly checkingmy assumptions. —Rasmus Lerdorf
Resilience, Not “Quality”Readable code.Reasonable test coverage.Sane architecture.Good debugging tools.An engineering culture that values refactoring.These are measureable goals.
Manual TestingIt doesn’t always look like you think it looks.Real-Time Monitoring is the new face of testing.
Watching The GraphsEtsy collects well over a quarter million metrics.Deciding which ones matter is a human problem.Everyone watches some subset of the graphs.Human vision excels at anomaly detection.
QA Happens When??Exploratory testing can be performed any time.Rigorous, scientiﬁc approach.Focus on customer satisfaction rather than a spec.Equally useful before or after a release.
Just Quality“Assurance” is a terrible word. Let’s discard it.Quality exists, we just can’t assure or prove it.
There Is No Such Thing As A Formal Proof Of Quality.Yet most of us would agree it exists.I propose that “customer experience” is a betterterm-of-art-than “quality.”Though there’s no formal proof for that either.
Exploratory TestingAddresses areas that Developer Testing can’t.Developer Testing validates assumptions.The Tester’s job is to invalidate assumptions.
Technology InformsCustomer ExperienceExploratory Testing requires an understanding ofthe ways in which a whole system is intended toserve a community of users.The problem space has as much to do withtechnology as it does with product requirements.
Most bugs, most of thetime, are easily nailed giveneven an incomplete butsuggestive characterizationof their error conditions atsource-code level. —Eric S. Raymond
Source, diffs, logs.If your QA Analysts don’t look at these, teach them.
Customer SupportYour customer support operators spend more timetalking to your users than anyone else.Customer Support interface with users asindividuals rather than as aggregate data.
Further Reading“How Google Tests Software,” James Whittaker (especially chapter 5)“Look At Your Data,” John Rausser“Optimizing For Developer Happiness,” Chad Dickerson“Outages, Postmortems and Human Error,” John Allspawhttp://en.wikipedia.org/wiki/Swiss_cheese_model“What Is Exploratory Testing?,” James Bach“How Many Eyeballs Tame Complexity,” ESR“The Timeless Way of Building,” Christopher Alexander