Successfully reported this slideshow.
Cloud Testing Bill of Rights Top 10 Decision Criteria for a Web & Mobile Application Testing Platform By Scott Barber, Chief Technologist, President & CEO, PerfTestPlus, Inc. Why a Cloud Testing Bill of Rights? My collaboration with When the governmental structure of the United States was initially put into place, SOASTA began in the sum- it did not include the Bill of Rights. While many of the Founding Fathers were far mer of 2011 when I was from pleased about the absence of explicit citizens’ rights in the Constitution, they asked to preview a beta ver- knew that if they didn’t get the basic governmental structure in place quickly, the sion of CloudTest Lite. The new country wouldn’t last long enough for citizens’ rights to matter. This sounds positive experience I’ve had very much like how development and IT groups were formed, with quick and dirty with both the product and organizational and governmental structures that made sense to someone in the the SOASTA team has en- moment. abled our work together to continue. This begs the question: In your development or IT group, do you feel like you have the right to voice dissatisfaction and organize to champion change? If your experi- If you are familiar with my work then you know that I what change we are championing.” don’t commonly write about vendors or tools aside from I can think of no techniques, ideas or practices related to testing, which require sup- deliberately neutral educa- port from management, that get shut down more quickly than those that include the tional pieces. cloud. The cloud is gaining a degree of credibility when it comes to operations, but when it comes to testing, there are a disproportional number of vociferous detrac- SOASTA has proven to be tors as compared to champions. I have to wonder if the detractors are being driven one of those rare vendors by fear: fear of change and fear of being rendered irrelevant by advancement. speak freely about without This observation leads me to see the Cloud Testing Bill of Rights as a series of violating what mom taught proposed revisions to commonly accepted software development practices. They me about refraining from ensure citizens of the nation of software development the freedom of expanded saying anything at all if you options and the right to pursue betterment of their current situation. These revi- don’t have anything nice to say. simplify application performance testing. Your Cloud Testing Rights I’m hard pressed to imagine many individuals who would object to having the right to pursue these items appropriate to the needs and context of their project, applica- tion or organization without fear of being labeled a troublemaker.© Soasta, Inc. 2012 1
II. Provides an elastic, globally distributed test environment on demand and within minutes III. Has dynamic and fully automated test grid provisioning IV. Has the ability to build and modify complex tests within hours V. Supports dynamic web and mobile technologies and protocols VI. Controls test sessions to start, pause, stop, modify, re-start and analyze VII. Has a single integrated view of performance and monitoring metrics VIII. Rapidly isolates performance issues in real time at any scale IX. Allows functional and load testing from lab to production X. Is affordable enough for everyday testing In case these rights don’t strike you as important or it’s not immediately obvious to you what problems they address, read on. Load Generation A cornerstone of a business creating value by adding application performance test- ing is the ability to exercise the application under test with realistic loads. “Realistic” doesn’t only refer to the ability to deliver the correct volume, but also to deliver that volume from the places where actual users will be using your application. The before deciding which tool to use for your application performance testing needs. - ternally and from an end-user’s perspective. Each of these perspectives add value that the other two don’t, meaning that failing to consider and account for each of these will ultimately lead to unwanted surprises. Internal load generation is the most effective option for early performance assess- ments and for continually assessing and tuning hardware and software that will live in your hosted environment. External load generation is necessary to determine of your site or application and geographic proximity to the data center. Taking mea- surements from a true end-user perspective, or “Last Mile” testing, is necessary to assess and tune the front-end aspects of the site or application and determine actual user experience. When it comes to load generation, in almost all situations you’ll want the ability to generate load both internally and externally. If you’re part of an organization that hosts every component internally and access to those sites or applications will be available only to folks on the same internal network, then there is no need for ex- ternal testing. © Soasta, Inc. 2012 2
There is rarely a need to generate load from actual client machines. Generating exter- nal load from the cloud while observing a sampling of end-user interactions is entirely - ready part of your overall testing program. II. Elastic, globally distributed test environment - ters is that you have the right to be able to generate realistic, production volume loads from geographies representative of your actual user base. While generating load ex- ternally from across the street is better than only being able to generate internal load, it’s not nearly as valuable as concurrently generating load from a variety of geographic locations. It’s far more challenging today to predict performance from a particular geography than - able to the producers of sites and applications to improve performance in a particular geography. Implementing a cloud-based technology for load generation means that you no longer have to wait for users to complain before becoming aware of a problem. Cloud-based load generation can enable you to assess performance by user location by clicking a couple of extra buttons and running a couple of extra of tests. III. Dynamic and fully automated test grid provisioning on demand and within minutes Very much related to the previous two rights, you have the right to mix and match the generation points for your load quickly and simply. Having the ability to generate load from anywhere isn’t enough, and isn’t unique to cloud-based solutions. I, for one, was doing that ten years ago. Of course, it always seemed to involve a month to ship hard- ware with the right software pre-installed to remote locations, or to talk people working of, coordinate network connectivity and the other hundred details. And all this before - gested such a solution today! Speed and Completeness enough to be relevant without needing to learn a new tool each time a new technology is implemented. Performance engineers have long been begging for rapid testing tools that can be used with a wider range of technologies. Today, traditional testing tools continue to take weeks and months and only test older technologies. Two rights speak to today’s performance testers not being condemned to telling those same war stories ten years from now.© Soasta, Inc. 2012 3
Clearly, this right is not exclusive to cloud-based performance testing technologies. Rather, you have the right to demand this from your tool or service provider no matter don’t have days, let alone weeks, to wait for performance assessments. They have the right to results in enough time to do something more useful than shrug, mumble something about wishing they’d known sooner and logging an action item for the next release. The only hope we have of providing results on their timeline is if we can build and modify realistic tests, complex tests, in hours or less. V. Support for dynamic web and mobile technologies and protocols One could say that we have the right to demand that our tools provide support for new technologies and protocols. At the time of this writing, the most challenging of those from a load generation perspective is related to asynchronous yet inter-related use of both client- and server-side communications within a single site or application. Next year, it may be something entirely different. What will remain the same, however, will be our desire to not purchase and learn a new tool or service due to a technological advance. Of course there will always be bleeding-edge technologies that tools don’t support. And when one of those technologies catches on, vendors need not charge us obscene amounts of money for updates. The fact that some vendors still expect their loyal cus- tomers to pay exorbitant fees to AJAX or SOAP support, protocols that crossed from have the right to be treated better than that. Control Following the ability to create and implement tests quickly, and building tests against leading technologies without purchasing new tools, tool extensions are the ability to performance test effort. VI. Control test sessions to start, pause, stop, modify, re-start and analyze There was a time when it was a rare tool that allowed expert performance testers to write custom code to give them the kind of detailed control they needed to maximize a tool so they can accomplish tasks like changing load levels on demand during test runs. There are plenty of tools on the market that come with this kind of control built in. This speaks to performance testers having the right to spend the majority of their time doing performance testing instead of working around tool limitations. Any organization disabling their performance testing program by forcing their testers to work around their tools today is spending far more money in terms of wasted time, unfound perfor-© Soasta, Inc. 2012 4
mance issues and under-tuned sites and applications than it would cost to switch to a tool or service that just works. We most certainly deserve the right to ask our compa- nies to stop wasting money. Analytics When I was starting out as a performance tester, virtually all performance analysis was into the same spreadsheet of data that was manually exported from the load genera- tion tool to manually create charts and graphs so the team could manually analyze the modify the next test. VII. Single integrated view of performance and monitoring metrics Year after year my pleas for this capability fell on deaf ears. Today, most load gen- eration tools come with built-in monitoring. Many even come with real-time, integrat- ed monitoring on a single, customizable dashboard. To me, this is nothing short of a dream come true;; for any performance tester this is at least a right, if not an expecta- tion. Aside from saving an unbelievable amount of time, the increase in value per test rights, it’s the right to have monitoring software that works as part of your performance testing program that matters. VIII. Rapidly isolate performance issues in real time at any scale - ect. Today, these tools are commonplace, affordable, approachable, and even easy to - ties. The ability to isolate performance issues in real time not only saves time, as the right’s title implies, it all but eliminates the “it’s your tool, not my application” response, enabling the team to get right to work on resolving issues instead of wasting time – The overwhelming majority of my clients’ performance testing efforts are completely hamstrung by tools, services and license agreements that make it impossible to do continual performance testing. This situation leads to the propagation of the belief that performance testing can be effective as a last-minute task. The reality is that treating performance testing as an isolated afterthought leads to unnecessary heroic efforts and frequently to epic failures in production. This is a sad and decidedly unhelpful state of affairs.© Soasta, Inc. 2012 5
IX. Test platform that allows functional and load testing from lab to production Good performance testing begins when a project is conceived and ends when a site or application is retired. Sites and applications that achieve consistent acceptable (or better) performance integrate performance testing into every aspect of design, devel- opment, deployment and maintenance of their product. It used to be that doing this took a large and expensive toolbox as well as a small army of tool specialists. That is no longer the case. With modern cloud-based performance testing tools, activities that used to be excep- tionally painful, like moving a test from the lab environment to a staging or production environment, take minutes instead of days or weeks. Combine this with capabilities of individual tools to perform multiple functions means that fully and seamlessly integrat- ing performance testing is entirely viable. Once again, we have the right to demand this kind of simple and seamless integration of performance testing into the entire product lifecycle and the right to demand that vendors support our efforts in this regard. X. Affordable enough for everyday testing This right simply cannot be overstated. For the majority of my career as a performance tester, only the richest and most elite software development organizations were able to justify the expense of buying or leasing the software and infrastructure necessary to run tests at will. Shared licenses, temporary environment provisioning, and administra- tive overhead has been the bane of a performance testers existence for years. Any organization that is still facing this situation is doing so out of negligence. Advances in cloud-based technologies and pricing models have all but eliminated the need for additional hardware and infrastructure to be procured and maintained in- ternally. This has reduced the total cost of enabling on-demand, daily performance testing, conducted by any member of any team, without the need to cross-coordinate hardware or license availability. Add that to the savings that an organization will realize by deploying and maintaining well-performing applications, and it seems nothing short of irresponsible to continue paying so much more for so much less value. Conclusion Cloud-based performance testing technologies have converted the dreams of many old-school performance testers into the rights of today’s performance testers. Failure to take advantage of these rights wastes time and money. It ensures acceptable site expensive, or result from sheer dumb luck. In short, not knowing and taking advan- tage of your rights as described by the Cloud Testing Bill of Rights is nothing short of irresponsible. About the Author: A prominent throught-leader in the area of software system performance and testing software systems, Scott Barber, founder and Chief Technologist of PerfTestPlus & Director of the Computer Measurement Group, makes his living writing, speaking, consulting, and coaching with the goal of advancing the understanding and practice of software testing. Scott has contributed to four books, composed over 100 articles and papers, the Association for Software Testing, and co-founded the Workshop on Performance and Reliability.© Soasta, Inc. 2012 6