Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mutation testing with PIT

32 views

Published on

Mutation testing with PIT

Published in: Software
  • Be the first to comment

  • Be the first to like this

Mutation testing with PIT

  1. 1. Mutation Testing with PIT Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
  2. 2. About me ● Integration Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
  3. 3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
  4. 4. I wrote code for the spacecraft.
  5. 5. How do you that it works?
  6. 6. Hmm...
  7. 7. I just know
  8. 8. I feel it
  9. 9. I feel it
  10. 10. I feel it
  11. 11. I feel it
  12. 12. $327.6 million
  13. 13. I wrote code for the spacecraft.
  14. 14. How do you that it works?
  15. 15. I wrote unit tests!
  16. 16. How do you know that your tests work?
  17. 17. Hmm...
  18. 18. really?
  19. 19. code
  20. 20. code test
  21. 21. code test test
  22. 22. code test test test
  23. 23. code test test test test
  24. 24. Does testing make any sense at all?
  25. 25. return a + b;
  26. 26. return a + b;
  27. 27. Coverage does not prove that you have TESTED every line. All it proves is that you have EXECUTED every line. And that is a big difference.
  28. 28. public class Calculator { public int sum(int a, int b) { return a + b; } }
  29. 29. @Test public void force100PercentCoverage() { calculator.sum(0, 0); }
  30. 30. Richard Lipton found the good answer in 1971
  31. 31. "If you want to know if a test suite has properly checked some code, introduce a bug"
  32. 32. return a + b;
  33. 33. return a - b;
  34. 34. Is it the whole idea behind the mutation testing?
  35. 35. Actually, Yes!
  36. 36. ● Artificial bug: MUTATION OPERATION
  37. 37. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT
  38. 38. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant
  39. 39. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant ● When the test succeeds on mutant: mutant SURVIVED
  40. 40. return a - b; KILLED SURVIVED
  41. 41. killing is good
  42. 42. My code is much more complex than adding two numbers!
  43. 43. Mutation Operators
  44. 44. Math + to - * to / - to + / to *
  45. 45. Boundary < to <= > to >= <= to < <= to >
  46. 46. Negate == to != >= to <
  47. 47. Complex Remove IFs
  48. 48. Complex Remove IFs Remove method calls
  49. 49. Complex Remove IFs Modify return Remove method calls
  50. 50. Complex Remove IFs Modify constantsModify return Remove method calls
  51. 51. Do I need to change the code on my own?
  52. 52. PIT Mutation Testing
  53. 53. public class Calculator { public int sum(int a, int b) { return a + b; } } @Test public void force100PercentCoverage() { calculator.sum(0, 0); }
  54. 54. DEMO
  55. 55. > MathMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
  56. 56. @Test public void Should_ReturnSum_When_SummingTwoValues() { // given int a = 1; int b = 2; // when int result = calculator.sum(a, b); // then assertThat(result).isEqualTo(3); }
  57. 57. > MathMutator >> Generated 1 Killed 1 (100%) > KILLED 1 SURVIVED 0 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
  58. 58. Do I need to read the console?
  59. 59. DEMO
  60. 60. How about bigger projects?
  61. 61. Hazelcast Kubernetes Plugin ● 5000 LOC ● 12 classes
  62. 62. DEMO
  63. 63. Why NASA engineers didn't use it in 1998?
  64. 64. 1971 Richard Lipton
  65. 65. 1971 Richard Lipton 2000
  66. 66. 1971 Richard Lipton 2000 2012
  67. 67. 1 Equivalent Mutants
  68. 68. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
  69. 69. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
  70. 70. 2 Slow Performance
  71. 71. return a + b;
  72. 72. return a + b;
  73. 73. return a + b; return a - b return 0
  74. 74. return a + b; return a - b return 0
  75. 75. :)
  76. 76. What about my team?
  77. 77. Main class paradox
  78. 78. // given
  79. 79. // given // when
  80. 80. // given // when ?
  81. 81. What to do with people who write tests without assertions?
  82. 82. Do people really use mutation testing?
  83. 83. Test your tests
  84. 84. Thank You! Rafał Leszko @RafalLeszko rafalleszko.com

×