Tells the story how we introduced static analysis to complement unit testing & code review, with a mature codebase. Previous attempts at using static analysis failed because the tools would find many "errors" that were not really interesting (coding style, etc.). Also, the tools would find many errors that existed for a long time, and were not problems for customers. This time, we concentrated on 2 things: - new code only (nightly scans - fix code that was checked in yesterday) - errors that result in run-time error Results: - Adoption by our developers - Better code quality - Developers wrote more resilient code the first time.

1. Introducing Static Analysis in a Mature Codebase
John Ruberto

2. Born out of failure

Telcom EMS/NMS system: 6M LOC C++




Very successful in the market
400K issues found through static analysis
Many false positives, each issue became a debate
Financial Web App: 1M LOC Java and javascript




Very successful in the market
20K issues found through static analysis
Each one to be reviewed, owner identified, communicated
Effort to manage the pile became overwhelming

3. How we did it







The case for change
What can static analysis do for you?
Process Design
Tool Selection
Deployment
Change Management
Backlog Reduction

4. Why use Static Analysis?

Root Cause Analysis: defects found in system test
Coding: Browser
Differences
Design:
4%
Functional
4%
Requirements:
Wrong
3%
Coding: Missing
6%
Design: Usability
6%
Content
Localization
1%
Coding errors
39%
Requirements:
Missing
9%
Missing
Code
9%
Coding:Exception
Handling
19%

5. How to improve our coding practices

Brain storm with developers:



Improve our unit test & code review practices
Introduce Static Analysis
According to Jones*, 97% of defects can be removed
by using these practices:

Code Review, Static Analysis, Unit Test
* Jones, Capers, 2011, Software Quality in 2011, A survey of the State of
the Art.

6. What can Static Analysis do for you?


Example of a defect caught by Static Analysis
Potential Null Pointer Exception:
boolean foo = bar.status();
if (bar != null) {
…
// do something here, but too late

7. Process Design

Minimize feedback loop



Focus on new issues
Alert developers privately



Creation -> Detection
Focus on developer productivity
Track issue status
Efficient management of issues

8. Process Design

9. Process Design, iteration 2

10. Tool Selection

Tool Requirements





Effectiveness with our development language
False positive rate with our code base
Compatibility with our development environment
Ability to uniquely identify defects
Either a built-in defect management system, or the ability
to integrate with our system through an API

11. Tool Evaluation Criteria

Efficacy of the checkers



False positive rate


How powerful the tool is for finding interesting defects
Interesting = defects that would result in runtime error
Tool indicates a defect, where there is no defect
False negative rate

Tool doesn’t find a defect, which indeed exists

12. Tool Evaluation - Results

Two tools with complimentary results: Coverity &
Findbugs

13. Deployment

Integrated into our build system
Initial scans: establish baseline
Scripting for issue assignment & notification
Training
Follow up, fixing glitches, etc.

4 weeks of effort





14. Automated Issue Management


Each nightly scan results in new issues & issues that
no longer appear
For each new issue:




Look up the developer that checked in that file
Assign the issue to the developer
Notify via email, with link to issue in the tool
For issues no longer appearing in the scans

Mark them fixed in the defect system

15. Weekly Reports


Regular communication is important for people to see the
value, and keep the change alive
Focus on action

16. Monthly Reports


Focus on Return on Investment
Reinforce the “new code” focus
1000
900
800
700
600
500
Found
Fixed
400
300
200
100
0
1/3/08
2/3/08
3/3/08
4/3/08
5/3/08
6/3/08

17. Backlog Reduction
First, build confidence in tool & issues
Then, let the games begin



During the Olympics:





10 pts each major
5 pts each moderate
1 pt each minor & each
close/no fix
Awards:
Gold, Silver, Bronze

Lottery:



1 “ticket” for each fix
Regular drawings
Regular posting of
winners & leaderboard
Image: Creative Commons: http://www.flickr.com/photos/garryknight/

18. Results: 0 backlog
Focus on new issues first, lead to confidence in the
tools and effort to eliminate the backlog
Static Analysis Issue Status
4500
4000
3500
3000
Isues

2500
Total
2000
Fixed
1500
1000
500
0
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep

19. Results: better code

Developers learn the anti-patterns: learn to write
more resilient code
Defects Detected by Static Analysis
250
200
150
100
50
0
Month 1
Month 2
Month 3
Month 4
Month 5
Month 6

20. Results: better quality

Better experience for customers


Improved Code Quality


Fewer failures occurring in production
> 97% - 100% of issues fixed before customer release
Reduced Technical Debt

Eliminated over 3000 issues

21. Five Tips





Engage Developers through the entire process (after
all, we are doing this for developers)
Choose the right tool for your codebase
Focus on new code, let “legacy” follow
Focus on developer productivity, not finding fault
Automate issue management

Triage, Assignment, Verification, Closure

22. Summary of benfits





100% code coverage, every night
Common errors found cheaply, supplements code
review and unit test
Feedback loop results in better coding skills
Identifies the exact line of code with the error
Objective closure criteria, its fixed or not fixed.

23. Q&A

Contact me:



JohnRuberto@gmail.com
Blog.ruberto.com
@JohnRuberto