Delta debugging is a technique for isolating the cause of failures in programs. It works by systematically removing parts of changes made between a version where the program worked and a version where it fails. The algorithm efficiently determines the minimal set of changes responsible for the failure. The technique was able to isolate a single change causing a failure in GDB from 178,000 changed lines within a few hours. Automating debugging through techniques like delta debugging can help developers quickly find and fix bugs.

1. Debugging Debugging
Andreas Zeller

2. Debugging

3. Debugging
Tracking

4. Debugging
Tracking
Simplifying

5. Debugging
Tracking
Simplifying
Automating

6. Debugging
Tracking
Simplifying
Automating
Fixing

7. Tracking
Simplifying
Debugging
Automating
Fixing

8. Tracking Debugging

9. 1997

10. 1997

11. 1997

12. 1997

13. 1997

14. 1997

15. 1997

16. 1997

17. 1997

18. 1997

19. 1997

20. Gregor Snelting

21. Vom Fachbereich f¨ r Mathematik und Informatik
u
der Technischen Universit¨ t Braunschweig
a
genehmigte Dissertation
zur Erlangung des Grades eines
Doktor-Ingenieurs (Dr.-Ing.)
Andreas Zeller
Configuration Management with Version Sets
A Unified Software Versioning Model
and its Applications
1. April 1997
1. Referent: Prof. Dr. Gregor Snelting
2. Referent: Prof. Dr. Walter F. Tichy
Eingereicht am: 1. November 1996

23. Bug Reports
From: Brian Kahne <bkahne@ibmoto.com>
To: DDD Bug Report Address <bug-ddd@gnu.org>
Subject: Problem with DDD and GDB 4.17
When using DDD with GDB 4.16, the run command
correctly uses any prior command-line arguments, or
the value of "set args". However, when I switched to
GDB 4.17, this no longer worked: If I entered a run
command in the console window, the prior
command-line options would be lost. [...]

35. Henry Lieberman

43. Simplifying Debugging

44. Bug Reports
From: Brian Kahne <bkahne@ibmoto.com>
To: DDD Bug Report Address <bug-ddd@gnu.org>
Subject: Problem with DDD and GDB 4.17
When using DDD with GDB 4.16, the run command
correctly uses any prior command-line arguments, or
the value of "set args". However, when I switched to
GDB 4.17, this no longer worked: If I entered a run
command in the console window, the prior
command-line options would be lost. [...]

45. Bisection

46. Bisection

47. Bisection
✔
Yesterday

48. Bisection
✔ ✘
Yesterday Today

49. Bisection
✔ ✘
Yesterday Today

50. Bisection
✔ ✔ ✘
Yesterday Today

51. Bisection
✔ ✔ ✘
Yesterday Today

52. Bisection
✔ ✔ ✘ ✘
Yesterday Today

53. Bisection
✔ ✔ ✘ ✘
Yesterday Today

54. Bisection
✔ ✔ ✔✘ ✘
Yesterday Today

55. Bisection
✔ ✔ ✔✘ ✘
Yesterday Today
Failure Cause

56. What was Changed
$ diff -r gdb-4.16 gdb-4.17
diff -r gdb-4.16/COPYING gdb-4.17/COPYING
5c5
< 675 Mass Ave, Cambridge, MA 02139, USA
---
> 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
282c282
< Appendix: How to Apply These Terms to Your New Programs
---
> How to Apply These Terms to Your New Programs
…and so on for 178,200 lines (8,721 locations)

57. Causality
Today
Program works
Program fails Yesterday
Cause

58. David Lewis
1941–2001

59. Causality
Today
Program works
Program fails Yesterday
Cause

60. Actual Causes
“The” cause (actual cause) is a minimal difference
Program works
Program fails
Actual cause

62. Today

63. Today
✘

64. Today
✘
Yesterday

65. Today
✘
✔
Yesterday

66. Today
✘
✔
Yesterday

67. Today
✘
✔
✔
Yesterday

68. Today
✘
✔
✔
Yesterday

69. Today
✘
✔
✔
Yesterday

70. Today
✘
✘
✔
✔
Yesterday

71. Today
✘
✘
✔
✔
Yesterday

72. Today
✘
✘
✔
✔
Yesterday

73. Today
✘
✘
?
✔
✔
Yesterday

74. Today
✘
✘
✔
✔
Yesterday

75. Today
✘
✘
✔
✔
Yesterday

76. Today
✘
✘
✔
✔
Yesterday

77. Today
✘
✘
?
✔
✔
Yesterday

78. Today
✘
✘
✔
✔
Yesterday

79. Today
✘
✘
✔
✔
Yesterday

80. Today
✘
✘
✔
✔
Yesterday

81. Today
✘
✘
✘
✔
✔
Yesterday

82. Today
✘
✘
✘
✔
✔
Yesterday

83. Today
✘
✘
✘
…
✘
✔
…
✔
✔
Yesterday

84. Today
✘
✘
✘
…
✘
✔
…
✔
✔
Yesterday

85. Today
✘
✘
✘
…
✘
✔
…
Failure Cause
✔
✔
Yesterday

86. Today test(c✘ ) = ✘
✘
✘
✘
…
∆ = c✘ c✔
✘
✔
…
Failure Cause
test(c✔
✔✔
)=
✔
Yesterday

87. test(c✘ ) = ✘ test(c✔ ) = ✔ ∆ = c✘ c✔

88. test(c✘ ) = ✘ test(c✔ ) = ✔ ∆ = c✘ c✔
dd(c✔ , c✘ ) = dd (c✔ , c✘ , 2)

89. test(c✘ ) = ✘ test(c✔ ) = ✔ ∆ = c✘ c✔
dd(c✔ , c✘ ) = dd (c✔ , c✘ , 2)
dd (c✔ , c✘ , n) =

(c✔ , c✘ )

 if |∆| = 1

dd (c ∆i , c , 2) if ∃i ∈ {1..n} · test(c✘ ∆i ) = ✔

✘ ✘



dd (c , c ∪ ∆ , 2) if ∃i ∈ {1..n} · test(c✔ ∪ ∆i ) = ✘



 ✔ ✔ i
dd c✔ ∪ ∆i , c✘ , max(n − 1, 2) else if ∃i ∈ {1..n} · test(c✔ ∪ ∆i ) = ✔

dd c , c ∆ , max(n − 1, 2) else if ∃i ∈ {1..n} · test(c✘ ∆i ) = ✘



 ✔ ✘ i

dd c , c , min(2n, |∆|)

else if n < |∆| (“increase granularity”)
✔ ✘



(c , c ) otherwise

✔ ✘

90. Delta Debugging
dd(c✔ , c✘ ) = dd (c✔ , c✘ , 2)
dd (c✔ , c✘ , n) =

(c✔ , c✘ )

 if |∆| = 1

dd (c ∆i , c , 2) if ∃i ∈ {1..n} · test(c✘ ∆i ) = ✔

✘ ✘



dd (c , c ∪ ∆ , 2) if ∃i ∈ {1..n} · test(c✔ ∪ ∆i ) = ✘



 ✔ ✔ i
dd c✔ ∪ ∆i , c✘ , max(n − 1, 2) else if ∃i ∈ {1..n} · test(c✔ ∪ ∆i ) = ✔

dd c , c ∆ , max(n − 1, 2) else if ∃i ∈ {1..n} · test(c✘ ∆i ) = ✘



 ✔ ✘ i

dd c , c , min(2n, |∆|)

else if n < |∆| (“increase granularity”)
✔ ✘



(c , c ) otherwise

✔ ✘

91. def dd(c_pass, c_fail):
n = 2
while 1:
delta = listminus(c_fail, c_pass)
deltas = split(delta, n); offset = 0; j = 0
while j < n:
i = (j + offset) % n
next_c_pass = listunion(c_pass, deltas[i])
next_c_fail = listminus(c_fail, deltas[i])
if test(next_c_fail) == FAIL and n == 2:
c_fail = next_c_fail; n = 2; offset = 0; break
elif test(next_c_fail) == PASS:
c_pass = next_c_fail; n = 2; offset = 0; break
elif test(next_c_pass) == FAIL:
c_fail = next_c_pass; n = 2; offset = 0; break
elif test(next_c_fail) == FAIL:
c_fail = next_c_fail; n = max(n - 1, 2); offset = i; break
elif test(next_c_pass) == PASS:
c_pass = next_c_pass; n = max(n - 1, 2); offset = i; break
else:
j = j + 1
if j >= n:
if n >= len(delta):
return (delta, c_pass, c_fail)
else:
n = min(len(delta), n * 2)

92. Isolating Changes

93. Isolating Changes
Delta Debugging Log
100000
GDB with ddmin algorithm
10000 ... with dd algorithm
... plus scope information
Changes left
1000
100
10
1
0 50 100 150 200 250 300
Tests executed
• Result after 98 tests (= 1 hour)

94. The Failure Cause
diff -r gdb-4.16/gdb/infcmd.c gdb-4.17/gdb/infcmd.c
1239c1278
< "Set arguments to give program being debugged when it is
started.n
---
> "Set argument list to give program being debugged when
it is started.n
• Documentation becomes GDB output
• DDD expects Arguments,
but GDB outputs Argument list

95. Andreas Zeller · TU Braunschweig
Gestern lief mein Programm.
Heute nicht mehr. Warum?
Fehlersuche mit Delta Debugging
Andreas Zeller
Mittagsseminar, TU Braunschweig, 16. M¨rz 1998
a
0

96. Yesterday, my program worked.
Today, it does not. Why?
Andreas Zeller
Universit¨ t Passau
a
Lehrstuhl f¨ r Software-Systeme
u
Innstraße 33, D-94032 Passau, Germany
zeller@acm.org
Abstract. Imagine some program and a number of changes. If none of these
changes is applied (“yesterday”), the program works. If all changes are applied
(“today”), the program does not work. Which change is responsible for the fail-
ure? We present an efficient algorithm that determines the minimal set of failure-
inducing changes. Our delta debugging prototype tracked down a single failure-
inducing change from 178,000 changed GDB lines within a few hours.
1 A True Story
The GDB people have done it again. The new release 4.17 of the GNU debugger [6]

102. Automating Debugging

103. VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac
System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION
VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System
Delta Debugging
8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System
8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS
X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION
VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION
VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION
VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION
VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION
VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/
1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION
VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT>
</td>
<td align=left valign=top>
<SELECT NAME="priority" MULTIPLE SIZE=7>
>
<OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION
VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT>
</td>
<td align=left valign=top
<SELECT NAME="bug_severity" MULTIPLE SIZE=7>
<OPTION VALUE="blocker">blocker<OPTION

104. VALUE="Mac System 7">Mac System 7<OPTION VALUE="Mac System 7.5">Mac
System 7.5<OPTION VALUE="Mac System 7.6.1">Mac System 7.6.1<OPTION
VALUE="Mac System 8.0">Mac System 8.0<OPTION VALUE="Mac System
8.5">Mac System 8.5<OPTION VALUE="Mac System 8.6">Mac System
8.6<OPTION VALUE="Mac System 9.x">Mac System 9.x<OPTION VALUE="MacOS
X">MacOS X<OPTION VALUE="Linux">Linux<OPTION VALUE="BSDI">BSDI<OPTION
VALUE="FreeBSD">FreeBSD<OPTION VALUE="NetBSD">NetBSD<OPTION
VALUE="OpenBSD">OpenBSD<OPTION VALUE="AIX">AIX<OPTION
VALUE="BeOS">BeOS<OPTION VALUE="HP-UX">HP-UX<OPTION
VALUE="IRIX">IRIX<OPTION VALUE="Neutrino">Neutrino<OPTION
VALUE="OpenVMS">OpenVMS<OPTION VALUE="OS/2">OS/2<OPTION VALUE="OSF/
1">OSF/1<OPTION VALUE="Solaris">Solaris<OPTION
VALUE="SunOS">SunOS<OPTION VALUE="other">other</SELECT>
</td>
<td align=left valign=top>
<SELECT NAME="priority" MULTIPLE SIZE=7>
>
<OPTION VALUE="--">--<OPTION VALUE="P1">P1<OPTION VALUE="P2">P2<OPTION
VALUE="P3">P3<OPTION VALUE="P4">P4<OPTION VALUE="P5">P5</SELECT>
</td>
<td align=left valign=top
<SELECT NAME="bug_severity" MULTIPLE SIZE=7>
<OPTION VALUE="blocker">blocker<OPTION

105. <SELECT>

108. Failure Cause

111. Ralf Hildebrandt Holger Cleve Martin Burger

114. Statistical Debugging

115. Jim Jones Ben Liblit

116. Omniscient Debugging

117. Bil Lewis

118. Dynamic Slicing

119. Andy Ko

120. Bug Mining

121. Tom Zimmermann Harald Gall Nachi Nagappan

122. Automatic Fixing

123. Westley Weimer

124. WINNER OF JOLT PRODUCTIVITY AWARD
ANDREAS ZELLER
WHY PROGRAMS FAIL
A GUIDE TO SYSTEMATIC DEBUGGING
SECOND EDITION

129. Fixing Debugging

143. Rob DeLine Gina Venolia

149. Contracts
set_hour (h: INTEGER) is
-- Set the hour from `h'
require
sane_h: 0 <= h and h <= 23
ensure
hour_set: hour = h
minute_unchanged: minutes = old minutes
second_unchanged: seconds = old seconds

150. Bertrand Meyer Gary Leavens

154. Scienti c Method

155. Scienti c Method
Hypothesis

156. Scienti c Method
Code
Hypothesis

157. Scienti c Method
Problem Report
Code
Hypothesis

158. Scienti c Method
Problem Report
Code
Hypothesis
Run

159. Scienti c Method
Problem Report
Code
Hypothesis
Run
More Runs

160. Scienti c Method
Problem Report
Code
Hypothesis Prediction
Run
More Runs

161. Scienti c Method
Problem Report
Code
Hypothesis Prediction Experiment
Run
More Runs

162. Scienti c Method
Problem Report
Code
Observation
Hypothesis Prediction Experiment + Conclusion
Run
More Runs

163. Scienti c Method
Problem Report Hypothesis is supported:
re ne hypothesis
Code
Observation
Hypothesis Prediction Experiment + Conclusion
Run
More Runs

164. Scienti c Method
Problem Report Hypothesis is supported:
re ne hypothesis
Code
Observation
Hypothesis Prediction Experiment + Conclusion
Run
Hypothesis is rejected:
More Runs create new hypothesis

165. Scienti c Method
Problem Report Hypothesis is supported:
re ne hypothesis
Code
Observation
Hypothesis Prediction Experiment + Conclusion
Run
Hypothesis is rejected:
More Runs create new hypothesis
Diagnosis

166. An Explicit Process

167. An Explicit Process
Hypothesis
Prediction
Experiment
Observation
Conclusion

168. An Explicit Process
Hypothesis The execution causes a[0] = 0
Prediction
Experiment
Observation
Conclusion

169. An Explicit Process
Hypothesis The execution causes a[0] = 0
Prediction At Line 37, a[0] = 0 should hold.
Experiment
Observation
Conclusion

170. An Explicit Process
Hypothesis The execution causes a[0] = 0
Prediction At Line 37, a[0] = 0 should hold.
Experiment Observe a[0] at Line 37.
Observation
Conclusion

171. An Explicit Process
Hypothesis The execution causes a[0] = 0
Prediction At Line 37, a[0] = 0 should hold.
Experiment Observe a[0] at Line 37.
Observation a[0] = 0 holds as predicted.
Conclusion

172. An Explicit Process
Hypothesis The execution causes a[0] = 0
Prediction At Line 37, a[0] = 0 should hold.
Experiment Observe a[0] at Line 37.
Observation a[0] = 0 holds as predicted.
Conclusion Hypothesis is confirmed.

190. Tracking Debugging Simplifying Debugging
Automating Debugging Fixing Debugging