Fehlmann and Kranich - Measuring tests using cosmic

Measuring Tests Using COSMIC
2°International Conferen
IT Data collection, Analysis and Benchma
Tokyo (Japan) - October 22,
Thomas M. Fehlmann, Zü
Eberhard Kranich, Duisb
Testing ICT Services
in the Cloud

2IT Confidence 2014 – October 22, 2014 http://itconfidence2014.wordpress.com
Measuring Tests
Using COSMIC
Goals of the presentation
G1. Understand COSMIC measurements in testing
G2. Free software testing from lines of code (LoC)
G3. Measure and benchmark software testing

l 1981: Dr. Math. ETHZ
l 1991: Six Sigma for Software Black Belt
l 1999: Euro Project Office AG, Zürich
l 2001: Akao Price 2001 for original contributions to QFD
l 2003: SwissICT Expert for Software Metrics, ICTscope.ch
l 2004: Member of the Board QFD Institute Deutschland – QFD Architect
l 2007: CMMI for Software – Level 4 & 5
l 2011: Net Promoter® Certified Associate
l 2013: Vice-President ISBSG
Dr. Thomas Fehlmann
l 1981: Dr. Math. ETHZ
l 1991: Six Sigma for Software Black Belt
l 1999: Euro Project Office AG, Zürich
l 2001: Akao Price 2001 for original contributions to QFD
l 2003: SwissICT Expert for Software Metrics, ICTscope.ch
l 2004: Member of the Board QFD Institute Deutschland – QFD Architect
l 2007: CMMI for Software – Level 4 & 5
l 2011: Net Promoter® Certified Associate
l 2013: Vice-President ISBSG

Eberhard Kranich
l Mathematics and Computer Science
l Emphasis on Mathematical Statistics
l Mathematical Optimization
l Theory of Polynomial Complexity of Algorithms
l Working at T-Systems International GmbH in Bonn, Germany
l Six Sigma Black Belt for Software Development
l Software Quality Assurance Manager
l Mathematics and Computer Science
l Emphasis on Mathematical Statistics
l Mathematical Optimization
l Theory of Polynomial Complexity of Algorithms
l Worked at T-Systems International GmbH in Bonn, Germany
l Six Sigma Black Belt for Software Development
l Software Quality Assurance Manager

What is a Defect?
l Defect = Behavior impacting expected or required functionality of software
è How many bugs?
è By counting the
size of defect repositories?
è By number of entries???

Software Testing as a Game
l Tester sees selected sequences in the UML
sequence diagram
l Tester can “walk” the data movements when
planning or executing tests
è Functionality becomes visible to the agile team
è Defects impacting functionality become visible
to testers
Other
Application
Other
Application
Some
Device
8.// Move some data
9.// Move some data
10.// Move some data
Other
Device

Functionality, Defect Size, and Defect Density
l What happens if data movements have
defects?
l Testers mark the data movement where a
defect has been detected
l Same Metric:
èè ISO/IEC 19761 COSMICISO/IEC 19761 COSMIC
Other
Application
Other
Application
Some
Device
8.// Move some data
Move some data
Other
Device
l Functional Size
è Number of Data Movements needed to implement all FUR
l Test Size
è Number of Data Movements executed in Tests
l Test Story
è Collection of Test Cases aiming at certain FURs
l Defect Count
è Number of Data Movements affected by some defect detected in a test story

Defects Density Prediction?
Now he counts the defects!
And counts and adjusts test size
By ISO/IEC 19761 COSMICISO/IEC 19761 COSMIC
Other
Application
Other
Application
Some
Device
8.// Move some data
9.// Move some data
Other
Device
How does he knowHow does he know
that he foundthat he found
all the defects?all the defects?

ISO/IEC Standard 29119 on Software Testing
Published as ISO/IEC 29119 (2013-07)
International Standard
Defines the Test Process
Calls for suitable Test Measures

ISO/IEC Standard 29119 on Software Testing
Organisational Test Process
Test Management Processes
Static Test
Processes
Test Planning
Test Monitoring
& Control
Test Completion
Organisational Test
Documentation
Feedback on
Organisational Test
Documentation
Test Plan Updates
Test
Plan
Test
Completion
Report
Test
Measures
Dynamic Test
Processes
Test Plan,
Control
Directives
Test Management
Processes
Test Plan,
Test Completion Report,
Test Measures
Test
Measures
Test Plan,
Control
Directives
Test Plan,
Control
Directives
© Tafline Murnane and Stuart Reid from ISO/IEC JTC1/SC7 WG26 Software Testing

The SW Testing Qualifications Board
l ISTQB
è 295’000 Certificates
è Iqnite Conferences (Sydney 2013)
l Third after ITIL and PMI
l Importance of Testing grows

But it’s Even Worse…
l What is Defect Density?
è Defects per KDLOC?
l What is Test Coverage?
è Code lines executed by some test case?

SW Testing and SW Metrics
l Counting practices for defect counting are undocumented
è “Number of Defects Found” per Stages / with Tests / etc.
è How do you count “Number of Defects”?
l Is it simply the number of entries in a defect repository?
è How can you avoid double reporting?
è Or make sure two defects are reported twice and not in a single report?
l A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is
under review: “Defect Measurement and Analysis Handbook”
è By European cooperation
è Important enhancement for ISBSG’s Data Collection!

SW Testing and SW Metrics
l Counting practices for defect counting are undocumented
è “Number of Defects Found” per Stages / with Tests / etc.
è How do you count “Number of Defects”?
l Is it simply the number of entries in a defect repository?
è How can you avoid double reporting?
è Or make sure two defects are reported twice and not in a single report?
l A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is
under review: “Defect Measurement and Analysis Handbook”
è By European cooperation
è Important enhancement for ISBSG’s Data Collection!
ReviewComments

Goal Profile
0.62
0.69
0.37
1) R001 Search Data
2) R002 Answer Questions
3) R003 Keep Data Safe
Functional User Requirements
A Simple Data Search Example
l Functional User
Requirements (FUR) describe
a very simple data search
l They meet Customer’s Needs
l And have a
Priority Profile
1 Entry (E) + 2 eXit (X) + 2 Read (R) + 1 Write (W) = 6 CFP
User DataData
1.// Search Criteria
Trigger
2.// Write Search
3.// Get Result
4.// Show Result
5.// Nothing Found
6.// Show Error Message
Customer's Needs Topics Attributes
Y.a Data Access y1 Access Data Always Reliable Frequently
y2 Repeatable Responses Responses identical Always
Y.b Data Integrity y3 Cannot impact data No Write allowed
Goal Profile derived fromGoal Profile derived from
Voice of the CustomerVoice of the Customer

The Following SW Tests look Appropriate:
l Test Stories (scenarios) have
è Many Test Cases
è Each Test Case has
• Test Data
• Known Expected Response
l Test Size and Test Profiles can be measured
è by Functionality Covered
Test Story
CT-A Prepare CT-A.1 Retrieve Previous Responses
CT-A.2 Detect Missing Data
CT-A.3 Data Stays Untouched
Priority
TestSize
FALSE
Measured Defect Profile
0.43 11
0.74 18
0.51 12
41
Show Defects

Execute Test Case CT-A.1.1
Entering valid search string
è Returns expected response
è Test Size is 4 CFP
Test Story No. 1
CT-A.1 Retrieve Previous Responses R001: Search Data R002: Answer Questions R003: Keep Data Safe Expected Response CFP
CT-A.1.1 Enter valid Search String X001,R001,W001,E001 X001,E001 R001,W001 Return (known) Answer 8
CT-A.1.2 Enter invalid Search String E001 R002,W001 Invalid Search String 3
Test Story Contribution (CFP): 5 2 4 Test Size 11
Test Case Measurements
for Test Story CT-A.1
User DataData
Trigger
2.// Write Search
3.// Get Result
4.// Show Result
5.// Nothing Found

Total Test Size
Total Test Size is 11 + 18 + 12 = 41 CFP
è Compares to Functional Size of 6 CFP
è Yields a Test Intensity of 41/6 = 6.8
è On average, < 7 tests per data movement
Test Story No. 1
CT-A.1 Retrieve Previous Responses R001: Search Data R002: Answer Questions R003: Keep Data Safe Expected Response CFP
CT-A.1.1 Enter valid Search String X001,R001,W001,E001 X001,E001 R001,W001 Return (known) Answer 8
CT-A.1.2 Enter invalid Search String E001 R002,W001 Invalid Search String 3
CT-A.2 Detect Missing Data R001: Search Data R002: Answer Questions R003: Keep Data Safe Expected Response CFP
CT-A.2.1 Enter valid Search String for No Data X002,R002,W001,E001 X001,R001,W001,E001 R002,W001 No Data Available 10
CT-A.2.2 Enter invalid Search String R001,W001,X002,E001 X002,E001 R002,W001 Invalid Search String 8
CT-A.3 Data Stays Untouched R001: Search Data R002: Answer Questions R003: Keep Data Safe Expected Response CFP
CT-A.3.1 Enter valid Search String W001,E001 X001,R001 R001,W001 Return identical Answer 6
CT-A.3.2 Enter invalid Search String X002,E001 Invalid Search String 2
CT-A.3.3 Enter Same String Again R001,W001,X001,E001 Return identical Answer 4
Test Size in CFP: 41
Test Intensity in CFP: 6.8
Test Coverage: 100%

Recording a Defect
Test Story No. 2
l “Bug” in 6.// Show Error Message
è Detected with data base containing no data
è Test Size is 4 CFP
è 1 Defect1 Defect1 Defect1 Defect found!
User DataData
Trigger
2.// Write Search
3.// Get Result
4.// Show Result
5.// Nothing Found

Defect Observed
One Defect Found
è Possibly observable in all tests touching the data movement 6.// Show Error Message – named X002
è Counted once
Name Label Description Name Label
#001 Escape Chars Some characters such as 'ä' are wrongly interpreted as escape characetrs in strings R001 GetResult
1
Defects Observed Data Movements Affected
Defect Count
Test Story No. 2

Test Status Reporting
l Test Status Characterization
è Test Size is the total number of data movements executed in all Test Cases of all test stories
è One Data Movement can have only one defect identified per Test Story
l However, one misbehavior found might affect more than one data movement and thus
count for more than one defect
Test Status Summary
Total CFP: 6
Defects Pending for Removal: 2 Test Size in CFP: 41
Defects Found in Total: 2 Test Intensity in CFP: 6.8
Defect Density: 33% Test Coverage: 100%

Caveat
l Test size increases with meaningless test cases added to the test stories – little variations
of test data with almost identical expected responses might spoil measurements!
è We need a metric indicating that our test strategy is appropriate

What is Six Sigma Testing?

What is Lean Six Sigma Testing?
Replace manyReplace many
Test StoriesTest Stories……
… by those
needed for the
Eigensolution
…… go for thego for the
Eigenvector!Eigenvector!

Calculating the Eigenvector with Jacobi Iterative
AAAAT
x

Calculating the Eigenvector with Jacobi Iterative
AAAAT
x9999 0 3 0
0 9999 0 8888
1 0 5 3
9999 0
0 9999
3 0
0 8888
90 0 24
0 145 24
24 24 35

Test Stories
GoalTestCoverage
RetrievePreviousResponses
DetectMissingData
DataStaysUntouched
AchievedCoverage
CT-A.1
CT-A.2
CT-A.3
R001 Search Data 0.62 5 8888 2 0.64
R002 Answer Questions 0.69 2 6 8888 0.66
R003 Keep Data Safe 0.37 4 4 2 0.40
Ideal Profile for Test Stories 0.43 0.74 0.51 Convergence Gap
0.42 0.7 0.5 0.04
0.10 Convergence Range
0.20 Convergence Limit
Test Stories
Deployment Combinator
Measuring Test Coverage with Eigensolution
Number ofNumber of
data movementsdata movements
executedexecuted

Lean & Six Sigma for Software Testing
Six Sigma
è Design of Experiments
è Multi-linear Regression
for Root Cause Analysis
and process control
l Lean
è Detect Waste (muda, 無駄
)
è Test-Driven Development
è Defect-free delivery
ll Lean Six SigmaLean Six SigmaLean Six SigmaLean Six SigmaLean Six SigmaLean Six SigmaLean Six SigmaLean Six Sigma
è Predict Waste (muda, 無駄)
è Eigenvectors solutionsEigenvectors solutionsEigenvectors solutionsEigenvectors solutions for Root
Cause Analysis and process control
è Predict Defect DensityDefect DensityDefect DensityDefect Density
è Q Control ChartsQ Control ChartsQ Control ChartsQ Control Charts for SW Testing

Questions?

Fehlmann and Kranich - Measuring tests using cosmic

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Fehlmann and Kranich - Measuring tests using cosmic

Similar to Fehlmann and Kranich - Measuring tests using cosmic (20)

More from International Software Benchmarking Standards Group (ISBSG)

More from International Software Benchmarking Standards Group (ISBSG) (9)

Recently uploaded

Recently uploaded (20)

Fehlmann and Kranich - Measuring tests using cosmic