Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Embedding Quality Engineering in DevOps pipeline

How quality engineering, in terms of continuous testing can come to rescue to organizations that face challenges in a DevOps environment? Read the full presentation to know more

  • Login to see the comments

Embedding Quality Engineering in DevOps pipeline

  1. 1. Embedding Quality Engineering in DevOps Pipeline
  2. 2. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Speakers Product Marketing Director, Sauce Labs clint@saucelabs.com Clinton Sprauve Test Automation Consultant, Aspire Systems jayanthi.jayaraman@aspiresys.com Jayanthi Jayaraman
  3. 3. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND About Aspire Global technology services firm with core DNA of Software Engineering Specific areas of expertise around Software Engineering, Digital Services, Testing and Infrastructure & Application Support Global presence across North America, Europe, APAC and Middle East Recognized 9 consecutive times as “Best Place to Work for” by GPW Institute Vertical focus on Independent Software Vendors, Retail, BFSI, Education 2600+ employees; 150+ active customers CMMI Maturity Level 3, ISO 9001:2008 and ISO 27001 : 2013 certified A Great Place to Work- 9 Glorious Wins in a Row Quality Maturity Level Technology Awards Forrester Rated Aspire one of the Top Continuous Testing Service Providers (2017) UK Benelux India Middle East Singapore US Netherlands Poland
  4. 4. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND About Sauce Labs – The Continuous Testing Experts Legacy of innovation: 10 years Comprehensive integrations with CI/CD platforms servers – Jenkins, Bamboo, Microsoft VSTS and others 3,000 customers run over 2 million tests everyday on our platform 500Enterprise Customers 10Years of Innovation 2.0MTests Run / Day Co-founder, Jason Huggins, invented Selenium and co-developed Appium
  5. 5. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Agenda o Importance of Quality Engineering in DevOps o Different Dimensions of Quality Engineering o How Continuous Testing can Make/Break a DevOps Environment o Implementing Continuous Testing in a DevOps Environment – Use Case o Tools for Continuous Testing
  6. 6. IMPORTANCE OF QUALITY ENGINEERING IN DEVOPS
  7. 7. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND How we Develop Software has Changed New Processes & Tools Drive Faster Releases Desktop/Packaged Apps Thick-Client Apps Desktop Web Responsive Web Web Apps Mobile Apps Annually Quarterly Monthly Weekly Daily On Demand Waterfall Fast Waterfall Continuous Integration Continuous Delivery Proprietary Single Stack On-Premise Open Source Best of Breed Cloud-based Speed of Releases Apps Process Tools
  8. 8. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Transformational Changes In Quality App works in few environments; lots of back end code One team runs tests Test at one stage at end of dev process Few long tests, run infrequently Expensive proprietary test frameworks App works in in 100s of OS/Browser/device combos; lots of front end code Many teams run tests Test at many stages throughout dev process Many short tests, run frequently Free Open Source frameworks o Comprehensive Web & Mobile Coverage o UI/Functional Testing o Training o Access for distributed teams o Visibility across teams o Speed o Reliability o Automated testing o Parallel testing o Training
  9. 9. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND The Need for Quality Engineering Quality Engineering - One of the key levers catalyzing the DevOps mission o Functional, Non-functional and User experience Always tested and certified before release to production o Strategized Test Automation Ensures automation wherever possible o Shift Left Approach Starts testing at an earlier stage instead of waiting for a stable code base o Automation and certification of the developed functionalities Enables faster testing cycle
  10. 10. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Shift Left From QA to QE Project Lifetime No.ofDefects QA LifecycleQE Lifecycle High occurrence of defects at the later part of the project life cycle High occurrence of defects at the early part of the project life cycle Shift Left Approach Shift Left Approach Quality Built-in Review Req, Architecture and Design Complete Test Automation Continuous Integration / Continuous Testing Be a voice of the customer Provide Actionable Data Development Testing Quality Engineering Principles
  11. 11. DIFFERENT DIMENSIONS OF QUALITY ENGINEERING
  12. 12. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND 3 Dimensions of Quality Engineering Various facets of Quality Engineering Non- Functional Testing Tools and ProcessesFunctional o Cloud-based Test Automation o Test Automation Framework o CI/CD o DevOps o Application o Data o API Automation o Resolution Tests o Mobile/Device testing o Performance o Security o Load o Multi-platforms
  13. 13. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Cloud-based Test Automation Enables automatic testing and result management Provisions infrastructure specific to testing Enables testing combinations of modules and products Manages failures and trigger appropriate action items Provides intelligence for the systems to scale and distribute Builds pipelines that span across all functions of development Cloud-based solution facilitates seamless validation of the automated test cases Sample Cloud Environments: Proprietary VM, TFS, Sauce Labs, etc.
  14. 14. TOOLS FOR CONTINUOUS TESTING
  15. 15. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Tools for Continuous Testing Functional & Regression Usability Performance Testing Mobile Test Automation Security Continuous Integration API Cloud Testing o JMeter o LoadRunner o Demonster o ANTS o NeoLoad o QALoad o WebLoad o Silk Performer o Selenium o Lean FT o Test Complete o Silk Test o Coded UI o Squish o UserFly o UserVue o Appium o Sauce Labs o M-eux o Test Complete o Xamarin o Device Anywhere o Monkey Talk o Soap UI o Soap UI Pro o Fitnesse o TestComplete o Rest-Assured o OWASP ZAP o Veracode o Fortify o QAInspect o Jenkins o Bamboo o VSTS o TFS o Sauce Labs
  16. 16. HOW CONTINUOUS TESTING CAN MAKE/BREAK A DEVOPS ENVIRONMENT
  17. 17. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Importance of Continuous Testing in DevOps Test Early & Often Throughout Your CI/CD Pipeline Test Operate OPSDEV Continuous Testing
  18. 18. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Continuous Testing – Paves the Way for Continuous Delivery o Automate your testing as much as possible o Automate more effective and efficient test cases which provides a wider test coverage o Start automating early in the lifecycle o Continuous Feedback o Early Defect Detection o Pipeline Visibility o Reliable and Scalable Resources o Go/No Go Decision for Builds o Continuous Code Check-Ins o Never check-In a broken/uncompiled Code o Dev-Tester Collaboration Continuous IntegrationContinuous Delivery Continuous Testing
  19. 19. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Continuous Testing – Best Practices Automated deployment Centralized code repository Automated build process Automated unit and integration tests Code pushes should be built on a CI server Implement faster build mechanism Testing in a clone of the production environment CD Practices
  20. 20. IMPLEMENTING CONTINUOUS TESTING IN A DEVOPS ENVIRONMENT– USE CASE
  21. 21. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Four Types of Testing in Pipeline Continuous Testing Requires a Comprehensive Solution Fast Feedback Small Tests Few Platforms Isolated GOAL Stable Build Deploy Ready Monitor / Debug APPROACH End to End Testing Few Platforms Integrated Full Suite of Tests All Platforms In Parallel Business Critical Tests Relevant Platforms Targeted DEVELOPMENT INTEGRATION PRE-RELEASE FEEDBACK
  22. 22. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Four Types of Testing in Pipeline Continuous Testing Requires a Comprehensive Solution Fast Feedback Small Tests Few Platforms Isolated GOAL Stable Build Deploy Ready Monitor / Debug APPROACH End to End Testing Few Platforms Integrated Full Suite of Tests All Platforms In Parallel Business Critical Tests Relevant Platforms Targeted DEVELOPMENT INTEGRATION PRE-RELEASE FEEDBACK Test on every Pull Request SAUCE LABS Automated Tests In Parallel Automated Tests In Parallel ”Live” Tests Bug Reproduction Any Platform
  23. 23. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND How To Apply To Your Pipeline CODEPLAN CODEPLAN CODEPLAN Team 1 Shared Delivery Pipeline Production BUILD o Integration test TEST o API testing o Regression testing o Performance testing DEPLOY o Smoke test o Security testing o User acceptance testing o Visual testing OPERATE o Production testing o Monitoring o Debugging M E R G E R E L E A S E Team 2 Team 3 Team 4 Local Dev Environments PLAN o Feature definition o Test authoring o Test driven development CODE o Unit test o Component test Fast Feedback Stable Build Deploy Ready FeedbackVerify Functionality
  24. 24. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND How To Apply To Your Pipeline: Mobile CODEPLAN CODEPLAN CODEPLAN Team 1 Shared Delivery Pipeline Production M E R G E R E L E A S E Team 2 Team 3 Team 4 PLAN o Feature definition o Test authoring o Test driven development CODE o Unit test o Component test Desktops Mobile Emulators & Simulators Mobile Real Devices BUILD o Integration test TEST o API testing o Regression testing o Performance testing DEPLOY o Smoke test o Security testing o User acceptance testing o Visual testing OPERATE o Production testing o Monitoring o Debugging Local Dev Environments
  25. 25. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND Quality Engineering & Continuous Testing Implementation ` Use Case Scenario: Cloud-based automation testing for an online assessment solution Test Automation Manual Testing
  26. 26. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND CI Best Practices Employed o 8 hours o 7 browser in parallel o Proceed to validate Integration suite on completion o 3 browsers in parallel - Test report from Sauce Labs - verified during every PR review Deployment/Release testing done in phased manner using Sauce Labs Sauce Labs - A part of Acceptance Process Smoke suite Regression suite Integration suite o 15 minutes or abort o 7 browser + 7 devices in parallel o Raise Showstopper if any test fails, else continue to regression
  27. 27. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND CI Components Sauce Labs WebDriver tests matrix Sauce on-demand plugin Sauce Labs test publisher Automation Repo Sauce Connect
  28. 28. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND CI Workflow DEV Version Control Build & Unit Test QA UAT PROD Commit Test Results Feedback Commit Full Regression Test Results Trigger Feedback Commit Trigger Test Results Feedback Trigger Commit Trigger UAT Testing Feedback Commit Trigger Test Results Feedback TriggerCommit CIPipeline
  29. 29. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND ROI Realized AutomationToolStack Selenium Web Driver Java (Scripting) Jenkins (CI) Sauce Labs (Cloud Execution) Rest Assured (API) TechnologyStack Web Application Responsive Design iOS, Android & Windows Mobile Browser Support Rest APIs 50 2400 300 50 1920 300 0 500 1000 1500 2000 2500 3000 Smoke Regression Integration Smoke Regression Integration Automated 50 1920 300 Tests# 50 2400 300 Automation Coverage Modules / Coverage Tests # # of Browsers Before Automation After Automation Test Duration Team Size Test Duration Team Size Smoke (100%) 50 14 (7 desktop + 7 devices) 12 hours 12 15 mins 10 Regression (80%) 2400 7 desktop combos 3 weeks 10 10 hours 4 Integration (90%) 300 3 desktop combos 2 days 4 3 hours 2
  30. 30. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND ROI Realized Befor Automation Smoke Regression Integration Team Size 12 10 4 1 Befor Automation, 12 Smoke, 10 Regression, 4 Integration, 1 0 2 4 6 8 10 12 14 Team Size (Post Automation) Time Duration to Automate Module Resources Duration Smoke 3 20 days Regression 10 8 months Integration 6 1 month Total Savings – Last 1 Year Module # of runs Time saved (Hrs) Smoke 48 856.8 Regression 24 22752 Integration 12 1500 Maintenance Effort Module Hours/Month Smoke 12 Regression 80 Integration 25 Duration:Pre-Automation(in Hrs) Duration:Post-Automation(in Hrs) Smoke 12 0.24 Regression 120 10 Integration 64 3 0 20 40 60 80 100 120 140 Turnaround Time(Pre/Post Automation) e e
  31. 31. ©1996-2018 Aspire Systems, Inc. NETHERLANDS | POLAND | SINGAPORE | US | UK | BENELUX | ME | IND ROI Realized Case CI Execution Test Suite Time Infrastructure VM/Physical Machines Smoke Suite 15 mins 7 Browser combinations in Parallel 7 Devices in Parallel 30 Regression Suite 8 Hours 7 Browser combinations in Parallel 70 To accomplish the task of completing the Smoke Test in 15 mins and Regression Test in 8 hours, a minimum of 100 VM/Physical machines would be needed. Setting up the CI for this combination using Physical machines or VMs is a tremendous task and the cost involved in setting up and running this infrastructure is represented below : 3 Yr. Total Cost of Ownership On-Premises Virtual Machines Cloud Server $3,078,750 $3,278,175 $1,446,685 Storage $ - $ - $ - Network $590,286 $429,304 $ - IT-Labor $ - $ - $ - Total $3,669,037 $3,707,479 $1,446,685 $ 5,000,000 $ 4,000,000 $ 3,000,000 $ 2,000,000 $ 1,000,000 $ 0 On-Premise Virtual Machines Cloud Server Storage Network IT Labor
  32. 32. Q & A maria.josephine@aspiresys.com
  33. 33. Thank You maria.josephine@aspiresys.com

×