データを解析しプロセス改善につかった事例

1. [Case study]
Utilize STLC data for Process Improvement
Jun 30th, 2020
Rajat Dayma
Service Quality Assurance Group
Leisure Product Dept.
Commerce Company
Rakuten, Inc.

2. 2
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

3. 3
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

4. 4
[Career Summary]
> I am from India having more than 13+ years of
experience in QA field
> Worked in Banking, Finance, Insurance domains
> Having around 7-8 years' experience working with
Japanese clients
> ISTQB Foundation level and JLPT N2 certified
> Worked on service and product based companies
both
Self-Introduction

5. 5
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

6. 6
Today’s Theme
QA Role = Product and Process Quality
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/

7. 7
Today’s Theme
QA Role = Product and Process Quality
Process Product
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/

8. 8
Today’s Theme
QA Role = Product and Process Quality
Process
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/
Today’s Goal

9. 9
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

10. 10
STLC Data
Plan TestDevelop
In the testing phase, we view human error as a bug.
As of June 2020, there are 277k of them recorded and many points of improvement
to learn from the results It means that you can.

11. 11
STLC Data
Plan TestDevelop
In the testing phase, we view human error as a bug.
As of June 2020, there are 277k of them recorded and many points of improvement
to learn from the results It means that you can.

12. 12
STLC Data
Plan TestDevelop
In the testing phase, we view human error as a bug.
As of June 2020, there are 277k of them recorded and many points of improvement
to learn from the results It means that you can.

13. 13
STLC Data
Plan TestDevelop
STLC
Database
277k+ results
In the testing phase, we view human error as a bug.
As of June 2020, there are 277k of them recorded and many points of improvement
to learn from the results It means that you can.

14. 14
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

15. 15
Metrix and analytics flow
Category Meaning Example
FUNCTION Do not work as per specification
• Spec says ‘A’ link should be clickable but in app its not
working
DEGRADE Function worker earlier but not working now properly
• In previous R1 execution “Button A” was working fine but in
R2 its not working as expected
REQUIREMENT Not mentioned or wrong explanation in specification
• “A” user should able to purchase only one type of ticket but
actually it should purchase all types of tickets PRD is wrong.
EXISTING Existing product bug
• QA found difference in PC an SP behavior but its expected
based on usage
UI Usability, Security, Performance issues
• If any page is taking more time to load then its treated as
usability issues
ENVIRONMENT Environment issues which affects for our test execution • Because of wrong deployment of STG we raise the ticket
INVALID QA misunderstood functionality • Spec completely misunderstood by QA
LATER It’s bug but won’t be fixed this release. Future release
• Valid bug wont be fixed in this release and considered for
future releases
NONFUNCTION （userability）（performance（security） • Performance or security related issues
OTHER_SERVICE CWD, Coupon, Salesforce, Dependent on other teams to resolve • Bugs which are dependent on other systems to solve

16. 16
STLC
Database
277k+ results
Metrix and analytics flow

17. 17
STLC
Database
277k+ results
Metrix and analytics flow

18. 18
STLC
Database
277k+ results
Metrix and analytics flow

19. 19
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

20. 20
STLC Data
TestDevelop
STLC
Database
We saw a lot of environmental issues that arose during the development phase and the testing phase.
So we decided to focus on that.

21. 21
STLC Data
TestDevelop
STLC
Database
Developer
We saw a lot of environmental issues that arose during the development phase and the testing phase.
So we decided to focus on that.

22. 22
STLC Data
TestDevelop
STLC
Database
Test
Environment
Developer
We saw a lot of environmental issues that arose during the development phase and the testing phase.
So we decided to focus on that.

23. 23
STLC Data
TestDevelop
STLC
Database
Test
Environment
Developer
3rd party
system
We saw a lot of environmental issues that arose during the development phase and the testing phase.
So we decided to focus on that.

24. 24
STLC Data
TestDevelop
STLC
Database
Test
Environment
Developer
3rd party
system
We saw a lot of environmental issues that arose during the development phase and the testing phase.
So we decided to focus on that.

25. 25
Proxy Issue
35 % of environment bug treatment time spent on
Among them, “Deployment issue” category takes up the most time.
Test
Environment
3rd party
system
3rd PartyTrouble
#ofissues
cumulativeratio
Developer
35%
Find the root cause

26. 26
Proxy Issue
Source: xxx
35 % of environment bug treatment time spent on
Among them, “Deployment issue” category takes up the most time.
Test
Environment
3rd party
system
Wrong
setting
3rd PartyTrouble
#ofissues
cumulativeratio
Developer
Deployment Issue is the most
35%
Find the root cause

27. 27
Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative
data.
Find the root cause

28. 28
Ticket Ticket Comment Hearing Note
https://ticket.server.com/ticket/browse/BUG-99848
Due to DB data difference.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Dev team
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYY
https://ticket.server.com/ticket/browse/BUG-88090
【Expected value 】 is correct.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XX
Environment settingissue
Dev team inserted
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66427
Deployment for group
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Updating content
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-88726
it is environment
problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.
Deploytimingissue
but unfortunately
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-77062
All the following functions are modified by both our project
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
Deploytimingissue
Issue detail)
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66345
this is environment issue.
Because
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Deploytimingissue
Issue detail)
Multiple
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYYYYYYYYYYYYYYYYYYYYYYY
Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative
data.
Find the root cause

29. 29
Ticket Ticket Comment Hearing Note
https://ticket.server.com/ticket/browse/BUG-99848
Due to DB data difference.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Dev team
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYY
https://ticket.server.com/ticket/browse/BUG-88090
【Expected value 】 is correct.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XX
Environment settingissue
Dev team inserted
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66427
Deployment for group
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Updating content
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-88726
it is environment
problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.
Deploytimingissue
but unfortunately
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-77062
All the following functions are modified by both our project
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
Deploytimingissue
Issue detail)
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66345
this is environment issue.
Because
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Deploytimingissue
Issue detail)
Multiple
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYYYYYYYYYYYYYYYYYYYYYYY
Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative
data.
Find the root cause

30. 30
Ticket Ticket Comment Hearing Note
https://ticket.server.com/ticket/browse/BUG-99848
Due to DB data difference.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Dev team
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYY
https://ticket.server.com/ticket/browse/BUG-88090
【Expected value 】 is correct.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XX
Environment settingissue
Dev team inserted
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66427
Deployment for group
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Environment settingissue
Updating content
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-88726
it is environment
problem.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.
Deploytimingissue
but unfortunately
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-77062
All the following functions are modified by both our project
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
Deploytimingissue
Issue detail)
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
https://ticket.server.com/ticket/browse/BUG-66345
this is environment issue.
Because
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Deploytimingissue
Issue detail)
Multiple
YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYYYYYYYYYYYYYYYYYYYYYYY
Once the target of the problem has been determined, we gather qualitative data through interviews as well as quantitative
data.
Find the root cause

31. 31
Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution
It's to reflect on the economic value or not.
Calculate Return on Investment

32. 32
Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution
It's to reflect on the economic value or not.
Calculate Return on Investment

33. 33
Test
Environment
Developer Tester
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

34. 34
Test
Environment
Developer Tester
①Confirm & Create bug ticket
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

35. 35
Test
Environment
Developer Tester
①Confirm & Create bug ticket
②Investigate & Re-Deploy
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

36. 36
Test
Environment
Developer Tester
①Confirm & Create bug ticket
②Investigate & Re-Deploy
③Re-Test
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

37. 37
Test
Environment
Developer Tester
①Confirm & Create bug ticket
②Investigate & Re-Deploy
③Re-Test
①Confirm Create bug ticket : X hours
②Investigate & Re-Deploy : Y hours
③Preparation & Re-Test : Z hours
#of Dep issues / yar
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

38. 38
Test
Environment
Developer Tester
①Confirm & Create bug ticket
②Investigate & Re-Deploy
③Re-Test 300+
hours
Lost
①Confirm Create bug ticket : X hours
②Investigate & Re-Deploy : Y hours
③Preparation & Re-Test : Z hours
#of Dep issues / yar
Currently we are losing around 300 hours efforts to solve deployment issue
Calculate Return on Investment

39. 39
Calculate ROI : Because the number of man-hours spent on solving the problem and the value of the solution
It's to reflect on the economic value or not.
Calculate Return on Investment

40. 40
Pick up any number of projects and test them for additional man-hours and effectiveness
Calculate Return on Investment

41. 41
Pick up any number of projects and test them for additional man-hours and effectiveness
Calculate Return on Investment
Project A Project B
Project C
Project D Project E
Project F
Project G Project H
Project I
Project J Project K
Project L
Project M Project N
Project O
Project P Project Q
Project R

42. 42
Pick up any number of projects and test them for additional man-hours and effectiveness
Calculate Return on Investment
Project A Project B
Project C
Project D Project E
Project F
Project G Project H
Project I
Project J Project K
Project L
Project M Project N
Project O
Project P Project Q
Project R

43. 43
We decided to target this category because this solution has no cost and some of LPD team has already implemented
strong solution about that.
Objective Key elements Root cause
Reduction of
environment
al issues
Deployment
issue
3rd PartyTrouble
Deploy timing issue
Environment setting issueProxy Related
Issue
Other
Solutions
Calculate Return on Investment

44. 44
We decided to target this category because this solution has no cost and some of LPD team has already implemented
strong solution about that.
Objective Key elements Root cause
Reduction of
environment
al issues
Deployment
issue
3rd PartyTrouble
Deploy timing issue
Environment setting issueProxy Related
Issue
Other
Solutions
Solution A $
Solution B $$$
Solution C $$
Calculate Return on Investment

45. 45
We decided to target this category because this solution has no cost and some of LPD team has already implemented
strong solution about that.
Objective Key elements Root cause
Reduction of
environment
al issues
Deployment
issue
3rd PartyTrouble
Deploy timing issue
Environment setting issueProxy Related
Issue
Other
Solutions
Solution A $
Solution B $$$
Solution C $$
Solution D $
Solution E $$$
Solution F $$
Calculate Return on Investment

46. 46
We decided to target this category because this solution has no cost and some of LPD team has already implemented
strong solution about that.
Objective Key elements Root cause
Reduction of
environment
al issues
Deployment
issue
3rd PartyTrouble
Deploy timing issue
Environment setting issueProxy Related
Issue
Other
Solutions
Solution A $
Solution B $$$
Solution C $$
Solution D $
Solution E $$$
Solution F $$
Calculate Return on Investment

47. 47
Set the target

48. 48
Source: xxx
Before After
35% 2.38%
Result

49. 49
Once proven effective, we will apply it to all projects.
Adaptation to the all projects
Project A Project B
Project C
Project D Project E
Project F
Project G Project H
Project I
Project J Project K
Project L
Project M Project N
Project O
Project P Project Q
Project R

50. 50
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement: deployment issue
7. Future
8. Conclusion
Agenda

51. 51
STLC
Database
277k+ results
Future
Of course, there are still many problems that can be solved with STLC database analysis.
So in the future we want to keep improving!
Branch merge issueEnvironment Isolation
issue
Requirement issue

52. 52
1. Self Introduction
2. Today’sTheme & Goal
3. What is STLC Data
4. a. Data collection and analysis
5. b. Metrix and analytics flow
6. Actual improvement deployment issue
1. Result 01
2. Result 02
3. Summary
7. Future
8. Conclusion
Agenda

53. 53
Conclusion
QA Role = Product and Process Quality
Process Product
https://www.kindpng.com/imgv/ihmmowT_this-free-icons-png-design-of-conveyor-with/