What we actually know about software development, and why we believe it's true.

1. Bits of Evidence What We Actually Know About Software Development, and Why We Believe It’s True Greg Wilson http://third-bit.com Feb 2010

2. Once Upon a Time... Seven Years’ War (actually 1754-63) Britain lost 1,512 sailors to enemy action... ...and almost 100,000 to scurvy

4. It Took a While to Catch On 1950: Hill & Doll publish a case-control study comparing smokers with non-smokers 1951: start the British Doctors Study (which runs until 2001)

5. What They Discovered #1: Smoking causes lung cancer “ ...what happens ‘on average’ is of no help when one is faced with a specific patient...” #2: Many people would rather fail than change

6. Like Water on Stone 1992: Sackett coins the term “ evidence-based medicine” Randomized double-blind trials are accepted as the gold standard for medical research The Cochrane Collaboration (http://www.cochrane.org/) now archives results from hundreds of medical studies

7. So Where Are We? “ [Using domain-specific languages] leads to two primary benefits. The first, and simplest, is improved programmer productivity... The second...is...communication with domain experts.” – Martin Fowler (IEEE Software, July/August 2009)

8. Say Again? One of the smartest guys in our industry... ...made two substantive claims... ...in an academic journal... ...without a single citation Please note: I’m not disagreeing with his claims —I just want to point out that even the best of us aren’t doing what we expect the makers of acne creams to do.

9. Um, No “ Debate still continues about how valuable DSLs are in practice. I believe that debate is hampered because not enough people know how to develop DSLs effectively.” I think debate is hampered by low standards for proof The good news is, things have started to improve

10. The Times They Are A-Changin’ Growing emphasis on empirical studies in software engineering research since the mid-1990s Papers describing new tools or practices routinely include results from some kind of field study Yes, many are flawed or incomplete, but standards are constantly improving

11. My Favorite Little Result Aranda & Easterbrook (2005): “Anchoring and Adjustment in Software Estimation” “ How long do you think it will take to make a change to this program?” Control Group: “ I’d like to give an estimate for this project myself, but I admit I have no experience estimating. We’ll wait for your calculations for an estimate.” Group A: “I admit I have no experience with software projects, but I guess this will take about 2 months to finish. ” Group B: “...I guess this will take about 20 months... ”

12. Results The anchor mattered more than experience, how formal the estimation method was, or anything else. Q: Are agile projects similarly afflicted, just on a shorter and more rapid cycle? Group A (lowball) 5.1 months Control Group 7.8 months Group B (highball) 15.4 months

13. Most Frequently Misquoted Sackman, Erikson, and Grant (1968): “Exploratory experimental studies comparing online and offline programming performance.” Or 10, or 40, or 100, or whatever other large number pops into the head of someone who can’t be bothered to look up the reference... The best programmers are up to 28 times more productive than the worst.

17. ...Which Explains a Lot Pessimists: “If we tackle the hump in the error injection curve, fewer bugs will get to the expensive part of the fixing curve.” Optimists: “If we do lots of short iterations, the total cost of fixing bugs will go down.”

18. The Real Reason I Care A: I've always believed that there are just fundamental differences between the sexes... B: What data are you basing that opinion on? A: It's more of an unrefuted hypothesis based on personal observation. I have read a few studies on the topic and I found them unconvincing... B: Which studies were those? A: [no reply]

23. Another Personal Favorite Conway’s Law: A system reflects the organizational structure that built it. Meant as a joke Turns out to be true (Herbsleb et al 1999)

24. But Wait, There’s More! Nagappan et al (2007) & Bird et al (2009): Physical distance doesn’t affect post-release fault rates Distance in the organizational chart does No, really — shouldn’t our development practices be built around these facts?

26. Folk Medicine for Software Systematizing and synthesizing colloquial practice has been very productive in other disciplines…

27. How Do We Get There? 2007 2008 – 2009

28. The Book Without a Name Wanted to call the next one Beautiful Evidence , but Edward Tufte got there first “ What we know and why we think it’s true” (By the way, his book is really good) Knowledge transfer A better textbook Change the debate

29. A Lot Of Editing In My Future Jorge Aranda Tom Ball Victor Basili Andrew Begel Christian Bird Barry Boehm Marcelo Cataldo Steven Clarke Jason Cohen Rob DeLine Khaled El Emam Hakan Erdogmus Michael Godfrey Mark Guzdial Jo Hannay Ahmed Hassan Israel Herraiz Kim Herzig Barbara Kitchenham Andrew Ko Lucas Layman Steve McConnell Audris Mockus Gail Murphy Nachi Nagappan Tom Ostrand Dewayne Perry Marian Petre Lutz Prechelt Rahul Premraj Dieter Rombach Forrest Shull Beth Simon Janice Singer Diomidis Spinellis Neil Thomas Walter Tichy Burak Turhan Gina Venolia Elaine Weyuker Laurie Williams Andreas Zeller Tom Zimmermann

30. The Hopeful Result

31. The Real Reason It Matters

32. Thank you, and good luck