This document discusses mutation testing and introduces Mull, a mutation testing tool for LLVM. It begins with an overview of software quality techniques like formal verification, fuzz testing, and unit testing. It then explains how mutation testing works by modifying code and running tests against the mutated code. The document outlines challenges with mutation testing like performance and generating too many mutants. It presents Mull as a way to address these challenges through techniques like smart mutant selection and runtime compilation. The document demonstrates Mull's use on test suites for LLVM and shows mutation testing scores. It concludes with open questions around integration, user experience, and supporting other languages.

1. Mutation Testing
Leaving the Stone Age
2017

2. whoami
• iOS Developer by day
• compiler hacker by night
• https://twitter.com/1101_debian
• https://lowlevelbits.org
• https://systemundertest.org

3. Outline
• Quality of Software
• Unit Testing
• Mutation Testing
• Mull
• Showcase: LLVM Test Suite

4. Quality of Software

5. Quality of Software
• Formal Verification

6. Quality of Software
• Formal Verification
• Fuzz Testing

7. Quality of Software
• Formal Verification
• Fuzz Testing
• Unit Testing + Code Coverage

8. Quality of Software
• Formal Verification
• Fuzz Testing
• Unit Testing + Code Coverage

9. Unit Testing

10. Unit Testing
int sum(int a, int b) {
return a + b;
}

11. Unit Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}

12. Unit Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}
Failed Tests: 0
Code Coverage: 100%
Passed Tests: 1

13. Mutation Testing
run_test(program, test)

14. Mutation Testing
run_test(program, test)
mutant = mutate(program)

15. Mutation Testing
run_test(program, test)
mutant = mutate(program)
result = run_test(mutant, test)

16. Mutation Testing
run_test(program, test)
mutant = mutate(program)
result = run_test(mutant, test)
if (result == Failed)
report_killed_mutant(mutant, test)

17. Mutation Testing
run_test(program, test)
mutant = mutate(program)
result = run_test(mutant, test)
if (result == Failed)
report_killed_mutant(mutant, test)
else
report_survived_mutant(mutant, test)

18. Mutation Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}

19. Mutation Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}
int sum'(int a, int b) {
return a * b;
}

20. Mutation Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}
int sum'(int a, int b) {
return a * b;
}
int sum''(int a, int b) {
return a - b;
}

21. Mutation Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}
int sum'(int a, int b) {
return a * b;
}
int sum''(int a, int b) {
return a - b;
}
test passed ->
mutant survived

22. Mutation Testing
int sum(int a, int b) {
return a + b;
}
void test() {
assert(sum(5, 10) > 0);
}
int sum'(int a, int b) {
return a * b;
}
int sum''(int a, int b) {
return a - b;
}
test passed ->
mutant survived
test failed ->
mutant killed

23. Mutation Testing
Killed Mutants: 1
Mutation Score = killed / total * 100%
Survived Mutants: 1
Total Mutants: 2
Mutation Score: 50%

24. Mutation Testing
• First proposed by Richard Lipton in 1971

25. Mutation Testing
• First proposed by Richard Lipton in 1971
• First implemented by Timothy Budd in 1980

26. Mutation Testing
• First proposed by Richard Lipton in 1971
• First implemented by Timothy Budd in 1980
• Studies say that MT was able to detect 70%-90%
of real faults

27. Mutation Testing
• Generates lots of data

28. Mutation Testing
• Generates lots of data
• Time consuming

29. Mutation Testing
• Generates lots of data
• Time consuming
• Languages are not mutation-testing-friendly

30. Mutation Testing
• Generates lots of data
• Time consuming
• Languages are not mutation-testing-friendly
• Problem of a Human Test Oracle

31. Mutation Testing
• Generates lots of data
• Time consuming
• Languages are not mutation-testing-friendly
• Problem of a Human Test Oracle
• "Excuse me, but I write good tests"

32. Mull

33. Mull
• Smart mutant selection

34. Mull
• Smart mutant selection
• Control over data generation

35. Mull
• Smart mutant selection
• Control over data generation
• Runtime compilation

36. Mull
• Smart mutant selection
• Control over data generation
• Runtime compilation
• Operates on LLVM IR level

37. Mull
• Smart mutant selection
• Control over data generation
• Runtime compilation
• Operates on LLVM IR level
• Language agnostic*

38. Mutant Selection

39. Mutant Selection

40. Mutant Selection

41. Mutant Selection

42. Mutant Selection

43. Mutant Selection

44. IRTests: 238 tests
Before:
391 modules
85 minutes

45. IRTests: 238 tests
Before:
391 modules
After:
85 minutes
124 modules
48 minutes

46. Mutation Control

47. Mutation Control

48. IRTests: 238 tests
Distance: 2
Number of mutants: ~1.5k
Real execution time: ~1 hour

49. IRTests: 238 tests
Distance: 2
Number of mutants: ~1.5k
Real execution time: ~1 hour
Distance: 29
Number of mutants: ~18k
Approximate execution time: ~11 days

50. Mutant Control

51. System Design

52. System Design
Mull

53. System Design
Core
• Driver
• Reporter
•
Mutation
Operators

54. System Design
Core
• Driver
• Reporter
•
Mutation
Operators
Toolchain
• JIT Compiler
• Object Cache

55. System Design
Core
• Driver
• Reporter
•
Mutation
Operators
Toolchain
• JIT Compiler
• Object Cache
Test Framework
• Test Finder
• Test Runner

56. System Design
Core
• Driver
• Reporter
•
Mutation
Operators
Toolchain
• JIT Compiler
• Object Cache
Test Framework
• Test Finder
• Test Runner
• Test Finder
• Test Runner
Google Test

57. System Design
Core
• Driver
• Reporter
•
Mutation
Operators
Toolchain
• JIT Compiler
• Object Cache
Test Framework
• Test Finder
• Test Runner
• Test Finder
• Test Runner
Google Test
• Test Finder
• Test Runner
XCTest

58. Showcase
• IRTests
• ADTTests

59. IRTests
Number of tests: 238
Number of mutants: 1.5k
Mutation score: 43%
https://lowlevelbits.org/IRTests/

60. ADTTests
Number of tests: 465
Number of mutants: 1.5k
Mutation score: 66%
http://lowlevelbits.org/ADTTests/

61. TripleTests.*
Number of tests: 13
Number of mutants: 624
Mutation score: 83%

62. TripleTests.*
T.setArch(Triple::mips64);
EXPECT_EQ(Triple::mips64el,
T.getLittleEndianArchVariant().getArch());

64. TripleTests.*
T.setArch(Triple::mips64);
EXPECT_EQ(Triple::mips64el,
T.getLittleEndianArchVariant().getArch());
T.setArch(Triple::tce);
EXPECT_EQ(Triple::tcele,
T.getLittleEndianArchVariant().getArch());
r294095, r294096

65. TripleTests.*
Triple T = Triple("");
T.setObjectFormat(Triple::ELF);
EXPECT_EQ(Triple::ELF, T.getObjectFormat());

66. TripleTests.*
Triple T = Triple("");
// T.setObjectFormat(Triple::ELF);
EXPECT_EQ(Triple::ELF, T.getObjectFormat());

67. TripleTests.*
Triple T = Triple("");
T.setObjectFormat(Triple::ELF);
EXPECT_EQ(Triple::ELF, T.getObjectFormat());
T.setObjectFormat(Triple::MachO);
EXPECT_EQ(Triple::MachO, T.getObjectFormat());
r294104

68. IRTests
if (foobar) {
fastVersion();
} else {
slowVersion();
}

69. Open Questions

70. Open Questions
• Integration

71. Open Questions
• Integration
• UX

72. Open Questions
• Integration
• UX
• Next Language

73. Open Questions
• Integration
• UX
• Next Language
• Many unknowns

74. Project: https://github.com/mull-project/mull
Contact: alex@lowlevelbits.org
Updates: https://twitter.com/1101_debian

75. Project: https://github.com/mull-project/mull
Contact: alex@lowlevelbits.org
Updates: https://twitter.com/1101_debian
Questions?