1.
Improve test automation operation
June 30th, 2020
Sadaaki, Emura
Leisure Product Dept.
Rakuten, Inc.

2.
2
Who am I
Name : Sadaaki Emura (nickname M)
Join in Rakuten : 2007
Career : Embedded engineer (2000-2002)
Web engineer (2002-2015)
Product manager (2015-2016)
QA (2016~)
Role: TestAutomation Engineer Lead
Birthplace : Kanazawa-city
Hobby: jog , climbing , horse racing

3.
3
Organization

4.
4
Organization
Developer group QA group
Manual test team automation test team
@IllustAC
ココ
の人
:

5.
5
Today story
• Test Automation operation is struggling
• Improve test automation operation

6.
6
Today story
• TestAutomation operation is struggling
• Improve test automation operation

7.
7
Test automation brings benefits in proportion to coverage!
• Reduce total cost of regression test
• Quick feedback
• Take work easy
© FREEIMAGES

8.
8
Test automation brings struggle actually
• Many failure reports come
• Synchronize latest specification
• a few engineers
©photoAC

9.
9
Classify many failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable

10.
10
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Good activity !
• Happy to report engineer!
©いらすとや

11.
11
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Script has bug
• Script covers old specification
• Modify script
©いらすとや

12.
12
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Test data invalid
• Scenario dependency issue
• Arrange test scenario
©いらすとや

13.
13
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Network disconnected
• 500 error (releasing operation?)
• 503 error (what’s happened?)
©いらすとや

14.
14
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable

15.
15
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
0.2 5.3
21.1
73.4
Fing bug Test automation script issue
Environment issue Temporary unstable

16.
16
Operation for Temporary unstable
1. Investigate failure report
2. Run test automation script again
3. Check if result is success
Ex. OperationCost
1 operation = 2 mins
Daily operation = 100 * 2 = 200 mins
Monthly operation = 200 * 22 = 75 hours
Waste time!

17.
17
Today story
• Test Automation operation is struggling
• Improve test automation operation

18.
18
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
0.2 5.3
21.1
73.4
Fing bug Test automation script issue
Environment issue Temporary unstable
• This failure account for ¾
• Recover operation is “Re-Run”

19.
19
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes

20.
20
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Test result output
• Console error message by Ranorex
• Error page screenshot

21.
21
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Use data
• Console error message
• Message in screenshot
• Screenshot
• Previous prediction data
UseTechnology (Python)
• Tesseract-OCR
• Deep Neural Network with Keras

22.
22
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
Input test report
Get message from
screenshot
Text base
prediction
Image base
prediction
No result
Output prediction
In progress
Feedback

23.
23
Auto recover system
Test Result is Failure
Predict reason
Temporary
unstable
Re run
Yes
• Reduce waste time operation
• Not waste test resource and time

24.
24
Auto recover system benefit
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20
auto recovery manual recovery
Ex.
Date : March-20
Temporary unstable Ratio : 75%
Operation time without system: 100 hours
100 * 0.65 = 65 hours saved
65%

25.
25
Summary
• Test automation operation takes time and includes boring task
• Let’s automate some of operation
• Auto recover system is not perfect yet, keep improvement