Empirical Software Engineering at Microsoft Research

Empirical Software Engineering
at Microsoft Research

Christian Bird
Nachiappan Nagappan
Brendan Murphy
Thomas Zimmermann, Microsoft Research

© Microsoft Corporation

The ESE Group at Microsoft
Socio-technical congruence
• Contribution behavior, technical relationships, and quality
• Does organizational structure affect bugs?
• Investigating the effects of geographic distribution

Bug reporting and triage
• What makes a good bug report?
• Characterizing bug reassignments and fixed bugs
• Collaboration and information needs in bug reports

Data-driven software engineering
• Failure-prediction/risk analysis
• Assessment of test-driven development and unit testing
• Analytics for software development


Collaborations of the ESE Group
Main locations of the ESM Group:
Redmond, Cambridge (UK)
Collaborations with other MSR Labs:
Microsoft Research India (Bangalore),
Microsoft Research Asia (Beijing),
European Microsoft Innovation Center
(Aachen, Germany)

Interns 2007-2010: U of Virginia (Ray
Buse); U of California, Santa Cruz (Ken
Hullett); National Institute of
Technology, Tiruchirappalli, India
(Kalaikumaran Ramamurthy); Stanford U
(Philip Guo); Boğaziçi U, Turkey (Ayse
Tosun); Darmstadt U of Technology
(Andreas Johansson); North Carolina
State U (Lucas Layman, Mei Nagappan)

Visitors 2007-2010: Hong Kong U of
Science and Technology (Sung Kim); U of
Zurich (Harald Gall, Martin Pinzger);
Saarland U (Andreas Zeller); North
Carolina State U (Laurie Williams); U of
Maryland (Victor Basili); Darmstadt U of
Technology (Neeraj Suri).


Outline
Socio-technical congruence
• Contribution behavior, technical relationships, and quality
• Does organizational structure affect bugs?
• Investigating the effects of geographic distribution

Bug reporting and triage
• What makes a good bug report?
• Characterizing bug reassignments and fixed bugs
• Collaboration and information needs in bug reports

Data-driven software engineering
• Failure-prediction/risk analysis
• Assessment of test-driven development and unit testing
• Analytics for software development


Nachiappan Nagappan, Brendan Murphy, Victor Basili @ ICSE 2008

INFLUENCE OF ORGANIZATIONAL
STRUCTURE ON SOFTWARE QUALITY


Motivation
• Conway’s Law:
– “Organizations that design systems are constrained to
produce systems which are copies of the
communication structures of these organizations.”
• Brooks argues in the Mythical Man Month:
– product quality is strongly affected by that structure.
• Little empirical evidence for relationship between
organizational structure and software quality


Factors that influence software quality

• Number of changes
• Number of people who make changes
• Number of people who left the company
• Cohesiveness of contributions
– Do most edits come from the team that owns
the component?
– How many organizations do make many edits
to the component?


Examples of organizational metrics
Total Edits = 250 Total Engineers Editing = 32 Total Ex-Engineers = 0

Top Level 30 Other Top 40 30
Manager M.L0 Level Manager M.L0 M.L0 10
200 40

M.L1 10 M.L1 M.L1 190 M.L1 M.L1 Org Size
No Of Edits
Ln = Level n
Owner (>75% edits) M.L2 190

E1 E2 E3 E4 E5 E6 E7
30 45 60 25 30

NOE = 32 NOEE = 0 EF = 250 DMO = 2 PO = 7/30
OCO
© Microsoft Corporation = 190/250 OOW = 5/32 OIF (>10% edits) = 2

Organization metrics
• The more people who touch the code, the lower the quality (NOE)
• A large loss of team members affects the knowledge retention and
lowers the quality (NOEE)
• The more edits to components, the higher the instability and the
lower the quality (EF)
• The lower the level the ownership, the better the quality (DMO)
• The more cohesive the contributors (organizationally), the higher the
quality (PO)
• The more cohesive the contributions (edits), the higher the quality
(OCO)
• The more the diffuse the contribution to a binary, the lower the
quality (OOW)
• The more diffuse the different organizations contributing code, the
lower the quality (OIF)


Defect prediction

Model
Software element Prediction
PCA
Regression
Metrics Bayes Classification
Churn

Dependencies Ranking

Org
…


Defect prediction

Model
Software element Prediction
PCA
Regression
Predictor Precision Recall Bayes
Metrics Classification
Pre-Release Bugs 73.80% 62.90%
Churn
Test Coverage 83.80% 54.40%
Dependencies
Dependencies 74.40% 69.90% Ranking
Code Complexity 79.30% 66.00%
Code Churn 78.60% …
79.90%
…
Org. Structure 86.20% 84.00%


Interpretation
Org B
Org A

Low Binary

Risk Bug Fix
New
Feature

Org D
Org C


Interpretation
Org B
Org A

Binary
Bug Fix
New
Feature

Org D
Org C
High Risk Known


Interpretation
Org B
Org A

Binary
Bug Fix
New
Feature

High Risk
Unknown
Org D
Org C


Christian Bird, Nachi Nagappan, Prem Devanbu,
Harald Gall, Brendan Murphy @ ICSE 2009

DOES DISTRIBUTED DEVELOPMENT
AFFECT QUALITY?


Distributed development
• Global development is becoming more common.
• There are many challenges, lots of research
– Multiple workshops, now a conference (ICGSE)
– Prior research enumerates many of the hurdles and effects on
time to market, code size, process difficulty
• What is the effect on code quality?

This study focuses on the
effect of distributed development on
software quality in Windows Vista


Data Timeline

Gather development and organizational data up to Vista RTM.
Measure code quality as post-release failures in the first 6
months following Vista’s release


Examining binaries
Level of distribution is the smallest entity that 75% of the
commits come from

Circles not proportional to
© Microsoft Corporation number of binaries

Examining binaries
Level of distribution is the smallest entity that 75% of the
commits come from


Effect on quality
Use linear regression to examine effect on
post-release failures
Model: Levels of distribution
Diff Diff Diff Diff Diff Number
Name Failures
Buildings Cafeterias Campuses Localities Continents of Devs

foo.dll 0 1 0 0 Variable
0 10 increase Std Err. Significance
% 3
(Constant) 0.09 p < 0.001
bar.exe 0 0 0 0 0 2 2
Diff Buildings 2.6% 0.42 p = 0.493
baz.sys 0 0 0 0 0 6 7
Diff Cafeterias 3.9% 0.18 p = 0.016
hello.exe 0 0 0 1 0 23 8
Diff Campuses 6.3% 0.29 p = 0.019
Diff Localities 8.3% 1.23 p = 0.457
Diff Continents -3.9% 0.26 p = 0.101
Num Devs 0.00 p < 0.001


Why did distributed development work?

• Not outsourcing
• Liaisons and face to face meetings
• Senior engineers started in Redmond
• Daily synchronous communication
• Consistent process and tools
• End to end ownership
• Common schedules
• Organizational integration


A confession…


Opinions expressed on this slide are the personal opinions of the presenter, not of Microsoft. ;-)


Backyard Safari Bug Vacuum Best Ever Bug Jar
Price: $19.82 Price: $8.12

Big Bunch O' Bugs
Price: $8.25

Watch a Bug
Price: $7.59
Found viaCorporation
© Microsoft

Incomplete bugs

Duplicate bugs


Too many bugs


BUG TRACKING 101


Your software fails…


Slide provided by Sascha
© Microsoft Corporation Just

Philip J. Guo, Thomas Zimmermann,
Nachiappan Nagappan, Brendan Murphy @ ICSE 2010

CHARACTERIZING WHICH BUGS
GET FIXED IN WINDOWS


Not all bugs get fixed
• Bug #1 occurs rarely and affects only a few users.
– Changes required to fix Bug #1 could be large and
expensive.

• Fixing Bug #2 could introduce new bugs
– Code changes that fix bugs are up to twice as likely to
introduce new bugs as other kinds of changes.

• Users are relying on existing behavior
– Fixing the bug could break their systems.


Research question

How do factors related to people and
bug report edits affect whether a bug
is successfully resolved as FIXED?

→ Inefficiencies in bug triaging process
→ Inform design of tools and policies


Bug handling is pervasive

24,000+ Microsoft employees
reported and handled Windows bugs

(vs. 2,000+ Windows developers)


Methodology

Qualitative Quantitative
survey data analysis

Reputation • Interest • Reassignments • Distance •
Additional qualitative factors from survey


Qualitative survey
• In your experience, how do each of these
factors affect the chances of whether a bug
will get successfully resolved as FIXED?
– 7-point Likert scale

• Sent to 1,773 Microsoft employees
– Employees who opened OR were assigned to OR
resolved most Windows Vista bugs
– 358 responded (20%)


SURVEY: How does the bug report's activity affect the
chances of it being successfully fixed?
Severity upgraded

Severity downgraded

Few editors (<10)

Many editors (>=10)

No re-assignments (0)

Few re-assignments (1-5)

Many re-assignments (>30)

At least 1 re-open

-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix


SURVEY: How do the people involved in addressing the
bug affect the chances of it being successfully fixed?
Opener/assignee same person

Same manager

Same building

Same country

Different managers

Different countries

Opened by temp. employee

Assigned to temp. employee

-3 -2 -1 0 1 2 3
Decreases chances of fix Increases chances of fix


SURVEY: Other qualitative factors
Textual quality of bug report:
“Just to re-emphasize: The quality of the bug description is very important. Not
necessarily filling in the dozens of fields in the bug database with all sorts of crap
(build numbers, dates, classifications, etc) - but just the plain-text description of
the problem, the implication and maybe even the potential solution.”

Perceived customer/business impact:
“Customer impact can be a very big impact on a bug if evidence exists to
show the cost of not fixing the bug for one or more customers.”

Rank and seniority of bug opener:
“A bug opened because something went wrong on a VPs [vice president’s]
laptop has better chance [of being fixed] than a bug opened because the
same thing happened to an intern.”

Interpersonal skills of bug opener:
“One other ‘soft’ factor is the speaking skill persuasiveness of the developer
(or other representative) when arguing for the bug.”


Quantitative data analysis
• Correlate likelihood of bug report getting fixed
– Reputation • Interest • Reassignments • Distance

• Windows Vista and Windows 7 bug reports
– Bug lifetime: open, assign, edit, resolve, close

• Microsoft employee data
– Manager, Building, Country


#1: Reputation

“A big influence [on bug fixes] is the
reputation of the person opening the bug.
If submitter has a history of submitting
high quality bugs then new bugs from that
person get better attention [...]”

From anonymous survey respondent


Quantifying reputation

For each bug, calculate opener’s reputation by
aggregating over all bugs in the past.

Hooimeijer and Weimer: Modeling bug report quality. ASE 2007.


Reputation vs. bug fixes


#2: Interest
Windows Vista


#3: Reassignments
Severity upgraded

Severity downgraded

Few editors (<10)

Many editors (>=10)

No re-assignments (0)

Few re-assignments (1-5)

Many re-assignments (>30)

At least 1 re-open

-3 -2 -1 0 1 2 3


#3: Reassignments
Windows Vista


The need for reassignments
“Bugs many times are exposed in the
UI [user interface], but are not caused
by the team writing the UI code.
These bugs can pass down several
layers of components before landing
on a lower level component owner.”



#4: Distance
Geographical

Organizational


Trust vs. distance

“Whenever I open bugs assigned to people
I know, they are investigated thoroughly as
there is a trust in the report I write.
Often when reporting a bug within areas
where I don't know the owners, there is
inherent distrust in the bug report.”



Recommendations
• Train and incentivize employees to write
higher-quality bug reports
• Improve awareness of developers’ expertise
to minimize reassignments
• Improve communication and trust amongst
people in different teams and locations
• Encourage more objectivity in prioritizing and
handling bugs


What metrics are the If I increase test coverage, will that
best predictors of failures? actually increase software quality?

What is the data quality level Are there any metrics that are indicators of
used in empirical studies and failures in both Open Source and Commercial
how much does it actually domains?
matter?
I just submitted a bug report.
Will it be fixed? Should I be writing unit
How can I tell if a piece
tests in my software
of software will have vulnerabilities? project?
Is strong code ownership good or
Do cross-cutting concerns bad for software quality?
cause defects?
Does Distributed/Global software
Does Test Driven Development (TDD) development affect quality?
produce better code in shorter time?


Ray Buse, Thomas Zimmermann @ FoSER 2010

TOWARDS ANALYTICS FOR
SOFTWARE DEVELOPMENT


Mining
Software
Repositories

Software
Analytics
Empirical
Software
Engineering


Mining
Software
Repositories

Software
Development
Analytics
Empirical
Software
Engineering


Analytics
“Use of analysis, data,
and systematic reasoning
to make decisions”
Financial services
Retail
Manufacturing
Health care
Energy
And more…


Analytics
Past Present Future

What What is What will
Information happened? happening now? happen?
(Reporting) (Alerts) (Extrapolation)

What’s the
How and why What’s the next
best/worst that
Insight did it happen? best action?
can happen?
(Modeling) (Recommendation)
(Prediction)

From Davenport et al. “Analytics at Work”.


Web analytics


Understanding data is hard!
The peak at
A is correct! So
Why is the
peak at B? what?
B
Researcher A
Developer


Each project is different!
Has to learn about
Knows how to analysis to understand
analyze data and act on results

Researcher Makes assumptions Knows project
about project very well Developer


Software development analyst

Knows enough
about data analysis

Knows enough
about project to
make the right
assumptions


Stakeholders have different needs
Researcher Developer Tester Dev Lead Test Lead Manager


A single tool is not enough

Surveys

Qualitative data
Measurements

Benchmarking


Make data actionable and accessible


#1: Data collection

Data focused

Integration
Scenario focused


#2: Data quality


#3: Privacy


#4: Understand user needs



Developers: well studied



Managers: not many studies




Managers: not many studies

Communication: not many studies



#5: User experience


#6: Education


Make data actionable and accessible

Data collection

Data quality Privacy
Education
Understand
user needs

User experience
http://msrconf.org

Empirical Software Engineering at Microsoft Research

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Empirical Software Engineering at Microsoft Research

Similar to Empirical Software Engineering at Microsoft Research (20)

More from Thomas Zimmermann

More from Thomas Zimmermann (20)

Recently uploaded

Recently uploaded (20)

Empirical Software Engineering at Microsoft Research