Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Improve test automation operation

1,221 views

Published on

自動化の結果データを自動化実行に使った話

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Improve test automation operation

  1. 1. Improve test automation operation June 30th, 2020 Sadaaki, Emura Leisure Product Dept. Rakuten, Inc.
  2. 2. 2 Who am I Name : Sadaaki Emura (nickname M) Join in Rakuten : 2007 Career : Embedded engineer (2000-2002) Web engineer (2002-2015) Product manager (2015-2016) QA (2016~) Role: TestAutomation Engineer Lead Birthplace : Kanazawa-city Hobby: jog , climbing , horse racing
  3. 3. 3 Organization
  4. 4. 4 Organization Developer group QA group Manual test team automation test team @IllustAC ココ の人 :
  5. 5. 5 Today story • Test Automation operation is struggling • Improve test automation operation
  6. 6. 6 Today story • TestAutomation operation is struggling • Improve test automation operation
  7. 7. 7 Test automation brings benefits in proportion to coverage! • Reduce total cost of regression test • Quick feedback • Take work easy © FREEIMAGES
  8. 8. 8 Test automation brings struggle actually • Many failure reports come • Synchronize latest specification • a few engineers ©photoAC
  9. 9. 9 Classify many failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable
  10. 10. 10 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Good activity ! • Happy to report engineer! ©いらすとや
  11. 11. 11 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Script has bug • Script covers old specification • Modify script ©いらすとや
  12. 12. 12 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Test data invalid • Scenario dependency issue • Arrange test scenario ©いらすとや
  13. 13. 13 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Network disconnected • 500 error (releasing operation?) • 503 error (what’s happened?) ©いらすとや
  14. 14. 14 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable
  15. 15. 15 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable 0.2 5.3 21.1 73.4 Fing bug Test automation script issue Environment issue Temporary unstable
  16. 16. 16 Operation for Temporary unstable 1. Investigate failure report 2. Run test automation script again 3. Check if result is success Ex. OperationCost 1 operation = 2 mins Daily operation = 100 * 2 = 200 mins Monthly operation = 200 * 22 = 75 hours Waste time!
  17. 17. 17 Today story • Test Automation operation is struggling • Improve test automation operation
  18. 18. 18 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable 0.2 5.3 21.1 73.4 Fing bug Test automation script issue Environment issue Temporary unstable • This failure account for ¾ • Recover operation is “Re-Run”
  19. 19. 19 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes
  20. 20. 20 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Test result output • Console error message by Ranorex • Error page screenshot
  21. 21. 21 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Use data • Console error message • Message in screenshot • Screenshot • Previous prediction data UseTechnology (Python) • Tesseract-OCR • Deep Neural Network with Keras
  22. 22. 22 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Input test report Get message from screenshot Text base prediction Image base prediction No result Output prediction In progress Feedback
  23. 23. 23 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes • Reduce waste time operation • Not waste test resource and time
  24. 24. 24 Auto recover system benefit 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20 auto recovery manual recovery Ex. Date : March-20 Temporary unstable Ratio : 75% Operation time without system: 100 hours 100 * 0.65 = 65 hours saved 65%
  25. 25. 25 Summary • Test automation operation takes time and includes boring task • Let’s automate some of operation • Auto recover system is not perfect yet, keep improvement

×