Unit testing ensures your production code is relevant. But what does ensure your testing code is relevant? Come discover mutation testing and make sure your never forget another assert again.
In the realm of testing, the code coverage metrics is the most often talked about. However, it doesn’t mean that the test has been useful or even that an assert has been coded. Mutation testing is a strategy to make sure that the test code is relevant.
In this talk, Nicolas will explain how Code Coverage is computed and what its inherent flaw is. Afterwards, he will describe how Mutation Testing work and how it helps pointing out code that is tested but leave out corner cases. He will also demo PIT, a Java production-grade framework that enables Mutation Testing.
3. @nicolas_frankel
Hazelcast
HAZELCAST IMDG is an operational,
in-memory, distributed computing
platform that manages data using
in-memory storage, and performs
parallel execution for breakthrough
application speed and scale.
HAZELCAST JET is the ultra fast,
application embeddable, 3rd
generation stream processing
engine for low latency batch
and stream processing.
4. @nicolas_frankel
• Unit Testing
• Integration Testing
• End-to-end Testing
• Performance Testing
• Penetration Testing
• Exploratory Testing
• etc.
Many kinds of testing
7. @nicolas_frankel
“Code coverage is a measure used to
describe the degree to which the
source code of a program is tested”
--Wikipedia
http://en.wikipedia.org/wiki/Code_coverage
Code coverage
7
8. @nicolas_frankel
Check whether a source code line is
executed during a test
• Or Branch Coverage
Measuring Code Coverage
@nicolas_frankel #mutationtesting #devexperience18
8
9. @nicolas_frankel
Computing Code Coverage
CC =
Lexecuted
Ltotal
*100
• CC: Code
Coverage
(in percent)
• Lexecuted: Number
of executed lines
of code
• Ltotal: Number of
total lines of
code
11. @nicolas_frankel
“Is 100% code coverage realistic? Of
course it is. If you can write a line of
code, you can write another that
tests it.”
Robert Martin (Uncle Bob)
https://twitter.com/unclebobmartin/status/55966
620509667328
100% Code Coverage?
11
13. @nicolas_frankel
• Any metric can be gamed!
• Code coverage is a metric…
⇒ Code coverage can be gamed
• On purpose
• Or by accident
Code coverage as a measure of quality
13
17. @nicolas_frankel
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
21. @nicolas_frankel
• Surviving means changing the
source code did not change the
test result Bad!
• Killed means changing the source
code changed the test result
Good!
Killed or Surviving?
21
22. @nicolas_frankel
Test the code
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
✔Execute Test
@Test
public void add_should_add() {
new Math().add(1, 1);
}
24. @nicolas_frankel
Test the code
✔Execute Test
public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
25. @nicolas_frankel
Killed mutant
✗Execute SAME Test
public class Math {
public int add(int i1, int i2) {
return i1 - i2;
}
}
@Test
public void add_should_add() {
new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
26. @nicolas_frankel
• PIT is a tool for Mutation testing
• Available as
• Command-line tool
• Ant target
• Maven plugin
Mutation Testing in Java
26
28. @nicolas_frankel
PIT mutators sample
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
36. @nicolas_frankel
• Analyze test code
• For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
Why so slow?
36
37. @nicolas_frankel
• Set a limited a set of mutators
• Limit scope of target classes
• Limit number of tests
• Limit dependency distance
• Increase number of threads
• Don’t bind to the test phase
• Use scmMutationCoverage
• Use incremental analysis
Workarounds
37
38. @nicolas_frankel
• Checks the relevance of your unit
tests
• Points out potential bugs
Is Mutation Testing the Silver Bullet?
38
39. @nicolas_frankel
• Validate the assembled application
• Integration Testing
• Check the performance
• Performance Testing
• Look out for display bugs
• End-to-end testing
• Etc.
The rest is up to you
39
40. @nicolas_frankel
• Don’t test to achieve 100%
coverage
• Test because it saves money in the
long run
• Prioritize:
• Business-critical code
• Complex code
Testing is about ROI
40