Call for Papers - Educational Administration: Theory and Practice, E-ISSN: 21...
A study on the efficiency of a test analysis method utilizing test-categories based on AUT and fault knowledge
1. A Study on the Efficiency of a Test
Analysis Method Utilizing Test-Categories
Based on AUT and Fault Knowledge.
15th Aug. 2014
ICACTCM’2014
Tsuyoshi Yumoto, Keiji Uetsuki, Tohru Matsuodani,
Kazuhiko Tsuda
2. Agenda
1. INTRODUCTION
2. VARIABILITY OF TEST ANALYSIS RESULTS
3. AN APPROACH OF TEST CATEGORIES BASED TESTING
4. REMARKS FROM THE VERIFICATION EXPERIMENT
5. CONCLUSION
2
3. 1. Introduction
Along with a rapid increase in the size and complexity of software
today, the number of required test cases is also increasing.
3
Testing
Development
・Requirement
・Specification
・Design
・Code Development
・Requirement
・Specification
・Design
・Code
increase in size
increase complexity
Testing
10 times
for 30 years
Quantity
FP Test cases
Quantity
FP Test cases
Test cases is equals the FP(function
point) sum to the power of 1.3 to 1.5
(by Capers Jones
4. 2. VARIABILITY OF TEST ANALYSIS RESULTS
Generally, a significant number of testers should be assigned to a project in
order to manage this increase of test cases.
Therefore, there are no clearly defined general rules for test development
process (see next page).
• They are developed according to the individual’s own judgment.
• This has the potential to cause the lacking or duplication of test cases.
Test analysis results when rules are not defined and interpretation is left to the individual
4
5. 2. VARIABILITY OF TEST ANALYSIS RESULTS(Cont.)
Definition of Test development process and Test analysis
• Testing performed at each level depicted in the V model has a process similar to
the development process, and three activities, test analysis, test design, and
test implementation in the test process are called Test development process.
5
A part of V-model
Test analysis
An activity of selecting
and organizing items to be
covered by the test .
6. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
A set of rules for a test analysis method for black box testing utilizing Test-
Categories based on the Application Under Test (AUT) and fault knowledge
have been proposed.
Overview of the method
The logical structure of a feature
Test –Category
The procedure and document format
6
7. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
The logical structure of a feature
• It can be used to test the feature in a MECE way . Each box in the logical
structure can be a useful guide to determine the required test conditions.
7
Logical Structure… a MECE way to find test conditions from features.
8. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
Test-Category
• In order to have a consistent interpretation of determining test conditions, a
8
name specialized for the AUT is put in each box of the logical structure.
• In order to ensure for clearly understanding the meaning of Test-Category ,
potential failures and/or faults which may arise for Test-Category are
discussed.
Knowledge of AUT
Knowledge of faults
experienced in the past
Building consensus
on the decided
Test-Categories
9. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
The procedure and document format
• Things used to determine test conditions
from a test basis are defined.
9
The outputs are eliminated by layering elements
based on the test case structure.
10. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
(Cont.)
The Main benefit of this method
• Higher test coverage overall, delivering higher quality testing
– By implementing the set of rules it will be easier to determine the necessary
10
test conditions.
Our hypothesis
The following issues currently make determining the necessary test conditions difficult:
1. Certain aspects of specification are not written if they are thought to be obvious.
2. Specification is not completely written within single target section in a document. (for
example: a behavior about a combination of functions)
– When many testers are involved in test development and proceed according to
the procedure and document format, all of the testers can carry out their work
according to the same set of rules.
The developed suite of test conditions are more
comprehensive and do not contain duplicates
11. 3. AN APPROACH OF TEST CATEGORIES BASED TESTING
(Cont.)
Another benefits of this method
• High Readability
– The procedure is based on the structure of the test conditions defined in the proposal of
11
this method . Elements which are included in the test basis are classified into
specification-items, expected results, and test parameters. One-by-one, these are
determined and selected through this procedure. Thus, the end result readability is
improved.
• Variation in the interpretation is minimized
– Once a consensus regarding the building of Test-Categories is met, team members can
more easily determine and select specification-items.
• Repeatable
– Since this method is structured, standardized and easy to walk though, testers will come
up with consistent and repeatable test analysis.
12. 4. REMARKS FROM THE VERIFICATION EXPERIMENT
The verification experiment Overview
Explain about
Test category
12
Results of the exercise conducted
Feature to be
tested
according to Test-Categories
Test
categories
Spec items Expected
results
Test Basis
Test
Parameters
Feature to be
tested
Spec Item Expected
results
Test
Parameters
Results
were
classified
to test
categories
Results of the exercise conducted
according to Test-Categories
1st exercise
2nd exercise
Comparison and Evaluation for Spec items
B - A A+
13. 4. REMARKS FROM THE VERIFICATION EXPERIMENT
(Cont.)
Evaluation results of the verification experiment
• Eight comparison results table were taken from the two verification experiments.
• There was a measurable improvement resulting from implementing the Test-
Categories method for seven out of eight teams.
• However, there was no indication as to exactly which categories received the
greatest benefit from implementing the method.
Music reproduction equipment Flight Booking Application
13
Logical Team
Structure
Logical
Structure
Team
TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2
Conv B A B B B B Conv A A
Input Input A B
Output - - - - - A+ Output A A
Storage - A+ - A+ A+ - Storage A A
Support B B B B B B Support B A
Mngt B A A A+ A A+ Mngt B A
14. 4. REMARKS FROM THE VERIFICATION EXPERIMENT
(Cont.)
Evaluation using patterns of input and output data of AUT
• In order to classify and evaluate the results of the verification experiments, patterns of
inputting and outputting data into/from the AUT (I/O data patterns) is focused as a
comprehensive AUT model.
The total combined patterns of
inputting and outputting data
into/from the AUT can be
summarized into the nine patterns
14
15. 4. REMARKS FROM THE VERIFICATION EXPERIMENT
(Cont.)
Evaluation results by I/O data patterns
Music reproduction equipment Flight Booking Application
15
Logical Team Logical
Team
Structure
Structure
TM1 TM2 TM3 TM4 TM5 TM6 TM1 TM2
Conv P1 - - - - - - Input P1 B B
Conv P1 B A+ B B B - Support P1 - A+
Support P1 B B B B B B Support P1 B B
Mngt P1 B A+ A+ A+ A+ A+ Storage P2 A+ A+
Storage P2 - A+ - A+ A+ - Input P4 - -
Conv P4 B B B B - B Mngt P4 B A
Mngt P4 B B B A+ B A+ Output P4 B B
Output P7 - - - - - A+ Output P4 A A
Input P7 A -
Conv P7 A A
Output P7 B B
Summarizing results in the I/O data pattern
P2 is 100% for both the
experiments. It can be said
that P2 has had the most
significant result.
There were no detailed
description about all spec items
for P2 in the test basis.
These results may suggest that
the results are aligned with our
assumption in Section 3.
Music reproduction equipment Flight Application
Evaluation level I/O Evaluation level
pattern
A+ A - B A+ A - B
I/O
pattern
P1 6 0 7 11 35.3% P1 1 0 1 4 20.0%
P2 3 0 3 0 100.0% P2 2 0 0 0 100.0%
P4 2 0 1 9 18.2% P4 0 3 2 3 50.0%
P7 1 0 5 0 100.0% P7 0 3 1 2 60.0%
the persentage = (A+ + A) /(A+ + A + B)
16. Conclusion
• Through these verification experiments it has been observed that after briefly
16
explaining this proposed method to participants there was a measurable
improvement in quantity and consistency of spec-item which they were able to
determine.
• Moreover, by analyzing I/O data patterns, the patterns of inputting and outputting
data into/from the AUT, a part of the evaluation results from the verification
experiments aligned with the assumption in the proposed method.
• Further verification experiments are necessary in order to carry out trend analysis
with higher accuracy.
• Conducting further experiments and deepening our understanding of the
tendencies and factors relating to effectiveness of the proposed method, rules for
creating Test-Categories based on the AUT knowledge and fault knowledge can be
more refined.
18. References
[1] C.Jones,"Estimating Software Costs 2nd Edition": McGraw-Hill, 2007.
[2] D.Longstreet, "Productivity of Software from 1970 to Present",
http://www.softwaremetrics.com/Articles/history.htm, 2000
[3] C.Ebert, and C.Jones. "Embedded Software: Facts, Figures, and Future." IEEE Computer
42.4 ,2009, pp 42-52.
[4] N.Owada"System integration 'straightforward way'",Nikkei Business Publications,
Inc,2009 (In Japanese).
[5] S. Eldh, H.Hansson, and S.Punnekkat, “Analysis of mistakes as a method to improve
test case design.” In Software Testing, Verification and Validation (ICST), 2011 IEEE Fourth
International Conference on.IEEE,2011,pp. 70-79
[6] T.Yumoto, T.Matsuodani, and K.Tsuda. "A Test Analysis Method for Black Box Testing
Using AUT and Fault Knowledge.": Procedia Computer Science 22 ,2013, pp.551-560.
[7] ISTQB FLWG, "Foundation Level Syllabus Version 2011": International Software Testing
Qualifications Board , 2011.
[8] T.J.Ostrand, and M.J.Balcer, "The category-partition method for specifying and
generating fuctional tests.":Communications of the ACM 31.6 ,1988,pp. 676-686.
[9] M.Grindal, and J.Offutt, "Input parameter modeling for combination strategies.":
IASTED International Conference on Software Engineering (SE 2007), 2007.
[10] G.J.Myers, C.Sandler, and T.Badgett, "The art of software testing ": Wiley, 2011.
18
19. References (cont.)
[11] B.Beizer, "Software testing techniques ": Dreamtech Press, 2003.
[12] Y.Nishi, "Viewpoint-based Test Architecture Design.": Software Security and Reliability
Companion (SERE-C), 2012 IEEE Sixth International Conference on. IEEE (2012), pp.194-
197.
[13] K.Akiyama, T.Takagi, and Z.Furukawa, "Development and Evaluation of HAYST Method
Tool (Software Testing)." : SoMeT. ,2010, pp.398-414
[14] S.Omura, "Phenomenology of general system": gohodoshoppan, 2005 (In Japanese).
[15] "IEEE standard for software test documentation ": IEEE829. 2002,2002
[16] Rasiel, E. M. The McKinsey Way. McGraw-Hill; 1999.
[17] Tsuyoshi Yumoto,Tohru Matsuodani,Kazuhiko Tsuda. "A practical Using Method for
Efficent Design of Functionai Testing": 75th National Convention of IPSJ; 2013 , paper#5B-4
(In Japanese).
[18] J.A.Whittaker, "How to break software ": Addison Wesley, 2003.
19
Editor's Notes
8月15日 11時15分~11時30分 12分、質問2分 12枚1分づつ
Good morning everyone. My name is Tsuyoshi Yumoto.
Today, I am going to talk about [A Study on the Efficiency of a Test Analysis Method that I have proposed. ]
This is an agenda of this presentation.
First of all, I will explain the background in section 1 and 2.
In short, I will talk about testing is the big challenge for software development
Actually, to solve these issues, I have proposed a method of Test Analysis Based on Test Categories. So, I will talk about what is test categories and how to use it in section 3
And then, this is the main topic of this paper, I will introduce remarks from the verification experiments for confirming the efficiency of this test analysis method.
This slide, I will tell you the situation of software development today. As I wrote in this slide, a set of test cases that required for software development is really increasing.
Actually, there is a survey that test cases is equals the function point sum to the power of 1.3 to 1.5.
That is why, the increase and complexity of software gives big impact to the quantity of test cases, and a significant number of testers should be assigned to a project in order to manage this increase of test cases.
Although the majority of testers are involved in the test development activity , we can see many software development projects that there are no clearly defined general rules for test development.
In these cases, testers are developed test cases according to the individual’s own judgment. This has the potential to cause the lacking or duplication of test cases.
By the way, I would like to tell you a definition of test development process and test analysis
Normally, testing has a process similar to the development process.
In the testing process, the test planning is performed in parallel with the activities on left side of the V model.
Then, another activities of test process are performed in chronological(クロノロジカル) order.
In addition, three activities, test analysis, test design, and test implementation in the testing process are also called test development process.
Because these are main activities to derive tests for AUT.
Underlying specifications to test design are artifacts on left side of the V model. These are called test basis.
When performing test development, it is necessary to refine the size of AUT at first.
In test development process, test basis is first input, and after analyzing and refining the test basis, and the determine what we should test. That is called test condition. test condition is going to be output of the activity. This activity is the test analysis.
To help solving this issue, I have proposed a set of rules for a test analysis method for black box testing utilizing Test-Categories based on the Application Under Test (AUT) and fault knowledge.
An overview of this test analysis method can be summarize three points.
One is, it is used the logical structure of a feature for determine test conditions. This is a reference model to see a feature in a MECE way . Each box in the logical structure can be a useful guide to determine required test conditions
Two is test category.
In order to have a consistent interpretation of determining test conditions using the logical structure as a guide, a name specialized for the AUT is put in each box of the logical structure of a feature. This is a test category.
And there is an activity in the procedure that potential failures and/or faults which may arise for Test-Category are discussed for building consensus among the testers of the project.
Three is, it is defined the procedure and document formats for test analysis activity.
We assume 2 changes by implementing the set of rules from the method. One is that it will be easier to determine the necessary test conditions. Two is that
all of the testers can carry out their work according to the same set of rules.
As a result, the developed suite of test conditions are more comprehensive and do not contain duplicates. This ensures higher test coverage overall, delivering higher quality testing. This is the main benefit of this method.
There are also another benefits to use this method.
One is High Readability. Two is minimized variation in the interpretation for test analyzing. three is improving to the repeatable process.
Every benefits are good for a project that has a lot of testers .
We have done two verification experiments to know the efficiency of test-category-based testing.
This is the exercise overview in the verification experiment.
We have got 8 comparison results from the two verification experiments. And there was a measurable improvement from implementation the test category method for seven out of eight teams.
But, there was no indication as to exactly which categories received the greatest benefit from implementing the test category method.
So, in order to classify and evaluate the results of the verification experiments as another way, We try to focus patterns of inputting and outputting data into/from the AUT as a comprehensive AUT model.
Because testing is the activity to do input data to AUT and get output data from AUT. It can be describe testing itself.
From this way , it is safe to sum up a few points.
P2 of patterns of inputting and outputting data into/from the AUT is 100% improving for both the experiments. It can be said that P2 has had the most significant result.
And there were no detailed description about all spec items for P2 in the test basis. These results may suggest that the results are aligned with our assumption in Section 3.
So, I would like to conclude this presentation.
We can observe a measurable improvement through the verification experiments. Besides this, the patterns of inputting and outputting data into or from the AUT, a part of the evaluation results from the verification experiments aligned with the assumption in the method that I proposed.
I think conducting further experiments and deepening our understanding of the tendencies and factors relating to effectiveness of the proposed method,
rules for creating Test-Categories can be more refined.
This is the end of my presentation.
Thank you very much.
This is the end of my presentation.
Thank you very much.
This is the end of my presentation.
Thank you very much.