Mutation testing (OOP 2012, 2012-JAN-24)

Mutation Testing

Filip van Laenen
OOP 2012
2012-01-24

© Computas AS 27.01.12

Agenda

• Basics of mutation testing
• Relation to other testing techniques
• Example
• Mutation testing techniques
• Mutation testing tools
• Personal experiences and recommendations
• Improvements
• Questions and comments

2 © Computas AS 27.01.12

Basics of
Mutation Testing


Mutation Testing in a Nutshell

Seeking The Summoner @ The Daily WTF
http://thedailywtf.com/Articles/Seeking-The-Summoner.aspx


Mutation Testing in a Nutshell (cont'd)



• Unit tests guard the source code
• But who guards the guardians?
• Do the unit tests cover all source code?
• Lines?
• Branches?
• Paths?
• Do the unit tests test the right things?

Mutation testing tests the tests!



def max(a, b) {
return (a < b) ? b : a;
}

def max(a, b) {
return (a ≤ b) ? b : a;
}


Does Mutation Testing Work?

“ In practice, if the software contains a
fault, there will usually be a set of
mutants that can only be killed by a test
case that also detects that fault.
Geist et. al., “Estimation and Enhancement of Real-time
Software Reliability through Mutation Analysis,” 1992


Does Mutation Testing Work? (cont'd)

“ Complex faults are coupled to simple
faults in such a way that a test data set
that detects all simple faults in a program
will detect most complex faults.
K. Wah, “Fault Coupling in Finite Bijective Functions,”
1995


Does Mutation Testing Work? (cont'd)

• “Generated mutants are similar to real faults.”
• Andrews, Briand, Labiche, ICSE 2005
• “Mutation testing is more powerful than
statement or branch coverage.”
• Walsh, Ph.D. Thesis, State University of New York at
Binghampton, 1985
• “Mutation testing is superior to data flow
coverage criteria.”
• Frankl, Weiss, Hu, Journal of Systems and Software,
1997


Relation to Other
Testing Techniques


Relation to Other Testing Techniques

• Unit tests
• Test-Driven Development (TDD)
• Test coverage
• Static code analysis
• Fuzz testing


Relation to Other Testing Techniques


Is Mutation Testing New?

• R. Lipton, “Fault Diagnosis of Computer
Programs,” 1971
• R. Lipton et. al., “Hints on Test Data Selection:
Help for the Practicing Programmer,” 1978
• Historical obstacles:
• No unit testing
• No TDD
• Time-consuming
• No integration with IDEs


Practical Example
of Mutation Testing


Practical Example

# Returns the maximum of a.
# @param a An array of integers.
def max(a) {
return …;
}


Practical Example (cont'd)

Omitted: max(null)
Omitted: max([])

def max(a) {
return …;
}



Assertion: max([0]) = 0

def max(a) {
return 0;
}




def max(a) {
return 0;
}




def max(a) {
return a.first;
}



Assertion: max([1, 2]) = 2

def max(a) {
return a.first;
}




def max(a) {
nit t ests!
m ← a.first; U age!
cover
foreach (e ∈ a) line
100% age?
if (e > m) over
nch c
m ← e;
100% b ra
return m;
}



def max(a) {
m ← a.first;
foreach (e ∈ a)
if (e > m)
m ← e;
return m;
}



def max(a) {
m ← a.first;
foreach (e ∈ a)
if (true)
m ← e;
return m;
}



def max(a) {
return a.last;
}




def max(a) {
m ← a.first;
foreach (e ∈ a)
if (e > m)
m ← e;
return m;
} 35 © Computas AS 27.01.12



def max(a) {
m ← -∞;
foreach (e ∈ a)
if (e > m)
m ← e;
return m;
} 36 © Computas AS 27.01.12



def max(a) {
m ← a.first;
foreach (e ∈ a)
if (e > m) ← Implicit else-branch!
m ← e;
return m;
}



def max(a) {
m ← -∞;
foreach (e ∈ a) ?
rage
if (e > m) h cove
← Implicitcelse-branch!
ran
m ← e; 10 0% b
return m;
}


• 100% test coverage may be illusory
• Line coverage
• Branch coverage
• Path coverage
• TDD principles easily broken
• Even if you're very careful


Mutation Testing
Techniques


Mutation Testing Techniques

• Three aspects:
• Mutation injection
• Mutation types
• Unit test selection per mutant
• Key properties:
• Efficiency
• Performance


Mutation Injection

• Source code mutation
• Binary code mutation
• Caveats:
• De-mutation
• Compilation errors
• Invalid binary code


Mutation Types

• Some mutations never change behaviour
• Constants reused by unit tests
• Log messages
• Some mutations can change behaviour
• Switching between < and ≠
• Switching between < and ≤
• Some mutations always change behaviour
• Switching between < and ≥


Mutation Types (cont'd)

def max(a, b) {
return (a < b) ? b : a;
}

def max(a, b) {
return (a ≤ b) ? b : a;
}

def max(a, b) {
return (a ≥ b) ? b : a;
}

Mutation Types (cont'd)

for (i ← 0; i < 10; i++) …

for (i ← 0; i ≠ 10; i++) …

for (i ← 0; i ≥ 10; i++) …


Mutation Types Guaranteed to Change
Behaviour *

• Negation of the comparison
• Switching between = and ≠
• Switching between < and ≥
• Switching between > and ≤
• Negation of boolean conditions
• Adding a ¬, ! or ~
• Shortcutting boolean conditions
• Replacement with True or False


Unit Test Selection

• Goal: find the unit test that “kills” the mutant
• Selection aids:
• Hints
• Name/package matching
• Code coverage tools
• Automatic learning
• Other heuristics


Unit Test Selection (cont'd)

• System:
• 50 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 50 × 20 × 1ms = 1s
• 10 mutants per class:
• Brute-force: 10 × 50 × 1s = 6m 20s
• Educated: 10 × 50 × 20 × 1ms = 10s


Unit Test Selection (cont'd)

• System:
• 500 classes
• 20 unit tests per class
• 1 ms per unit test
• Unit testing time: 500 × 20 × 1ms = 10s
• 10 mutants per class:
• Brute-force: 10 × 500 × 10s = 13h 53m 20s
• Educated: 10 × 500 × 20 × 1ms = 1m 40s


Complexity

• f: Number of function points
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × f²
• Educated force: τ × μ × φ × f
• Ideal: τ × μ × f


Complexity (cont'd)

• c: Number of classes
• φ: Number of function points per class, ≥ 1
• τ: Number of unit tests per function point, ≥ 1
• μ: Number of mutants per function point, ≥ 1

• Brute-force: (f × τ) × (f × μ) = τ × μ × φ² × c²
• Educated force: τ × μ × φ² × c
• Ideal: τ × μ × φ × c


Loops

for (i ← 0; i < 10; i++) …

for (i ← 0; i < 10; i--) …


Infinite Loops

• Terminate mutants that take too long to run
• What's too long?
• Ruins the performance
• Can be hard to predict


Other Problems

• Recursion:
• Stack overflows
• Out of memory exceptions
• Syntax errors
• Segmentation faults


Mutation Testing
Tools


Mutation Testing Tools

• Ruby: Heckle
• Java:
• Jester
• Jumble
• PIT
• C#: Nester
• Python: Pester


Heckle

• Ruby
• Test::Unit and rSpec
• Usually run from the command-line
• Runs a set of unit tests on a class or a method
• Good to-the-point reporting
• Good performance
• Virtually no documentation

• http://rubyforge.org/projects/seattlerb/
• http://docs.seattlerb.org/heckle/


Heckle Mutations

• Booleans
• Numbers
• Strings
• Symbols
• Ranges
• Regexes
• Branches (if, while, unless, until)


Heckle Sample Output

Initial tests pass. Let's rumble.
*****************************************************************
*** Greeter#greet loaded with 3 possible mutations
*****************************************************************

3 mutations remaining...
No mutants survived. Cool!


My Heckle Sample Output

filip@filip-laptop:~/github/wruf$ rake heckle
(in /home/filip/github/wruf)
Doing mutation testing on 15 method(s) of FlickrSearcher against
test/flickr_searcher_unit_test.rb:
o FlickrSearcher#convert_photo_info [1/15]
o FlickrSearcher#create_form_data_to_get_info_about_photo [2/15]
o FlickrSearcher#create_form_data_to_get_info_about_user [3/15]
o FlickrSearcher#create_form_data_to_search_photos [4/15]
o FlickrSearcher#do_rest_request [5/15]
o FlickrSearcher#get_author [6/15]
…
o FlickrSearcher#get_photo_info [12/15]
o FlickrSearcher#get_photo_url [13/15]
o FlickrSearcher#get_ref_url [14/15]
o FlickrSearcher#get_ref_url_from_xml_photo_info [15/15]
Checked 192 mutations, and no issues were found in FlickrSearcher.


My Heckle Sample Output (cont'd)

Doing mutation testing on 7 method(s) of WrufSettings against
test/wruf_settings_unit_test.rb:
o WrufSettings#dimensions [1/7]
o WrufSettings#dimensions= [2/7]
o WrufSettings#hours [3/7]
o WrufSettings#hours= [4/7]
o WrufSettings#tags [5/7]
o WrufSettings#tolerance [6/7]
o WrufSettings#tolerance= [7/7]
Checked 0 mutations, and no issues were found in WrufSettings.


Heckle Sample Report

--- original
+++ mutation
def calculate_precision(rescaled_number)
if (rescaled_number < 9.995) then
return 2
else
if (rescaled_number < 99.95) then
return 1
else
if (rescaled_number >= 99.95) then
return 0
else
- return -1
+ return -70
end
end
end
end

Jester

• Java
• JUnit
• Usually run from the command-line
• Grester for Maven2
• Operates on source code
• Runs all unit tests on all classes
• Reporting and documentation could be better

• http://jester.sourceforge.net/
• http://sourceforge.net/projects/grester/

Jester Sample Report Overview


Jester Sample Detailed Report


Pester and Nester

• Pester
• Jester for Python
• PyUnit
• Nester
• Port of Jester for C#
• NUnit
• Integrated with Visual Studio
• But outdated…
• http://nester.sourceforge.net/


Nester Sample Report


Jumble

• Java
• JUnit
• Run from the command-line
• Operates on byte code
• Runs unit tests on a class
• Reporting and documentation could be better
• Claims to be faster than Jester

• http://jumble.sourceforge.net/index.html


Jumble Sample Report

Mutating Foo
Tests: FooTest
Mutation points = 12, unit test time limit 2.02s
..
M FAIL: Foo:31: negated conditional
M FAIL: Foo:34: - -> +
......
Score: 67%


PIT

• Java
• JUnit
• Maven or command-line
• Operates on byte code
• Large set of mutators
• Also possibly equivalent mutators available
• Highly configurable
• Sensible defaults

• http://pitest.org/

PIT Mutators

• Conditionals Boundary
• Negate Conditionals
• Math
• Increments
• Invert Negatives
• Inline Constant*
• Return Values
• Void Method Call
• Non Void Method Call*
• Constructor Call*


PIT Sample Report


Personal Experiences
and Recommendations


Experiences and Recommendations

• Use mutation testing from day 1
• Start on a small code base
• Keep number of unit tests per class low
• Have small classes
• Select a good tool
• Configurable
• Flexible
• One that can output the mutant


Experiences and Recommendations
(cont'd)

• Believe the tool
• Or try to proof that the tool is wrong
• Fix the problem
• Don't turn mutation testing off
• Embrace the “more than 100%” test coverage
• Path coverage
• Less code
• More unit tests
• More intelligent unit tests


Improvements


Improvements

• Integration with more unit testing frameworks
• Better unit test–source code mapping
• Better heuristics
• Parallellisation
• Better reporting
• IDE integration
• Building tool integration


The Ideal Mutation Testing Tool™

• Easy integration with building tools
• Easy integration with IDEs
• Support for all unit testing frameworks
• Human-aided unit test selection heuristics
• Full parallellisation
• Good source code mutation reporting


Questions?

Contact:

fvl@computas.com @filipvanlaenen

Computas AS Tel +47-67 83 10 00
Lysaker Torg 45, pb 482 Fax +47-67 83 10 01
N-1327 Lysaker Org.nr: NO 986 352 325 MVA
NORWAY www.computas.com


Mutation testing (OOP 2012, 2012-JAN-24)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to Mutation testing (OOP 2012, 2012-JAN-24)

Similar to Mutation testing (OOP 2012, 2012-JAN-24) (20)

More from Filip Van Laenen

More from Filip Van Laenen (17)

Recently uploaded

Recently uploaded (20)

Mutation testing (OOP 2012, 2012-JAN-24)