Analytics for smarter software development

Analytics for Smarter
Software Development

Thomas Zimmermann
Microsoft Research, USA

© Microsoft Corporation

40 percent of major
decisions are based
not on facts, but on
the manager’s gut.
Accenture survey among 254 US managers in industry.
http://newsroom.accenture.com/article_display.cfm?article_id=4777


analytics is the use
of analysis, data, and
systematic reasoning
to make decisions.
Definition by Thomas H. Davenport, Jeanne G. Harris
Analytics at Work – Smarter Decisions, Better Results


software analytics: empower
software development teams to
gain and share insight
from their data to make better
decisions.
Raymond Buse, Thomas Zimmermann: Information Needs for Software
Development Analytics. ICSE 2012 SEIP Track.
http://research.microsoft.com/apps/pubs/default.aspx?id=172578

Smart analytics Usage analytics

© Microsoft Corporation © Microsoft Corporation

Development analytics The future

© Microsoft Corporation © Microsoft Corporation

Smart analytics


Jack Bauer

Chloe
O’Brian


All he needed was a paper clip

smart analytics is
actionable

smart analytics is
real time

Scene from the movie WarGames (1983).

smart analytics is
sharing

insight patterns data


smart analytics is
diversity

stakeholders tools questions
Researcher Developer Tester Dev. Lead Test Lead Manager


Clustering
Prediction Surveys

Qualitative Analysis
Measurements

Benchmarking
Segmenting

What-if analysis
Multivariate Analysis
Interviews


Build tools for
frequent questions

Use data scientists for
infrequent questions
Frequency

Questions


smart analytics is
people

The Decider The Brain The Innovator


inductive engineering

The Inductive Software Engineering Manifesto: Principles for Industrial Data Mining.
Tim Menzies, Christian Bird, Thomas Zimmermann, Wolfram Schulte and Ekrem
Kocaganeli. In MALETS 2011: Proceedings International Workshop on Machine
Learning Technologies in Software Engineering

Usage analytics


Improving the Explorer for Windows 8

Explorer
in Windows 7
Alex Simons: Improvements in Windows Explorer.
http://blogs.msdn.com/b/b8/archive/2011/08/29/improvements-in-windows-explorer.aspx



Customer feedback
• Bring back the "Up" button
from Windows XP,
• Add cut, copy, & paste into
the top-level UI,
• More customizable
command surface, and
• More keyboard shortcuts.


Overlay showing Command usage % by button on the new Home tab


Debugging in the (very) large
• Microsoft ships software to 1 billion users
– How do we find out when things go wrong?

• Fix bugs regardless of source application or
OS software, hardware, or malware
• Prioritize bugs that affect the most users
• Get the solutions out to users most efficiently
• Try to prevent bugs in the first place

K. Glerum, K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt:
Debugging in the (Very) Large: Ten Years of Implementation and Experience. SOSP 2009.


Windows Error Reporting

!analyze



billions Error reports collected
1 billion Machines run WER client code
100 million Reports /day processing capacity
many 1000s Bugs fixed
almost all Microsoft product teams use it
over 700 Companies using WER
200 TB of Storage
>60 Servers
>10 Years of use


Relative number of reports per bucket and cumulative distribution for
Top 20 Buckets from Office 2010 ITP for a 3 week sample period.

Project Gotham Racing 4
Across all races:
• 2 of 9 game modes were used
in < 0.5% of races
• 12 of 29 event types were used
in < 1% of races
• 50 of 134 vehicles were used
in < 0.25% of races

When looking at multiplayer races:
• 2 of 4 game modes were used
in < 2% of races
• 7 of 16 event types were used
in < 0.1% of races
• 53 of 133 vehicles were used
in < 0.25% of races

Kenneth Hullett, Nachiappan Nagappan, Eric Schuh, John Hopson:
Empirical analysis of user data in game software development. ESEM 2012.


Player progression in Halo 3

Bruce Phillips. Peering into the Black Box of Player Behavior: The
Player Experience Panel at Microsoft Game Studios. GDC 2010


Development analytics


SWEPT datamart
• Software Engineering Productivity Tools
• Set of data sources pertaining to product,
engineering process and organizations
• Provides consistency of data discovery and
access across product groups
• Provides a standard platform for creating and
deploying analytics
• Informs data driven decision making


Change analysis with CRANE
Risk
• How risky is the fix we are about to make?
• Which parts of the change are the riskiest?

Test
• Which subset of existing test cases should be executed to maximize
chances of finding defects?
• Which parts of the change will not be covered by existing tests and
need new tests?

Dependence
• What dependent parts of the system need to be re-tested?
• For code that exposes a public interface, which consumers of the
APIs should be verified?

Jacek Czerwonka, Rajiv Das, Nachiappan Nagappan, Alex Tarvo, Alex Teterev: CRANE: Failure Prediction,
Change Analysis and Test Prioritization in Practice - Experiences from Windows. ICST 2011.


1

2
3

4


5

6

7


Branches in Windows
networking

integration

main
integration
Process overhead
Time delay (velocity)
multimedia
Changes are isolated
=> Less build and test breaks
Christian Bird, Thomas Zimmermann:
Assessing the Value of Branches with What-if Analysis. FSE 2012.


Code flow for a single file

Orange nodes are
move operations

Blue nodes are
edits to the file


Branch decisions

How do we coordinate parallel
development?

How do we structure the branch
hierarchy? Can we reduce the
complexity of branching?


Assessing branches

Simulate alternate branch structure to assess cost
and benefit of individual branches

• Cost: Average delay increase per edit (liveness)
How much delay does a branch introduce into
development?

• Benefit: Provided isolation per edit (isolation)
How many conflicts does a branch prevent per edit?


Child Branch

Victim Branch

Parent Branch

Simulation (what-if)
Child Branch
faster
code flow

Victim Branch
unneeded
integrations removed
Parent Branch
no longer no longer no longer no longer no longer
isolated isolated isolated isolated isolated


Assessing branches
Red dots
are branches
with high cost
but low benefit

Delay
(Cost)

Each dot
is a branch Green dots
are branches
with high benefit
Provided Isolation
(Benefit)
and low cost


Assessing branches
Red dots
are branches
with high cost
but low benefit

If high-cost-low-benefit branches had been removed,
changes Delay each have saved 8.9 days of transit
would
(Cost)
time and only introduced 0.04 additional conflicts.
Each dot
is a branch Green dots
are branches
with high benefit
Provided Isolation
(Benefit)
and low cost


The future


INTELLIGENCE IN EVERYTHING

"The models I build are based on a
mix of social and computer
science, statistical data and my
constant travels around the world
talking to people about the future.
[…] How do we want to make the
lives of people all over the world
better by infusing our lives with
intelligence?"

 Brian David Johnson, Futurist, Intel

Photo via http://www.flickr.com/photos/intelfreepress/6793363054
Quote via Corporation
© Microsoft
http://mashable.com/2012/04/04/predictions-digital-future/

CLOUD BECOMES THE NORM

"My prediction is that the term
'cloud' will have disappeared from
the phrase 'cloud computing' by
2020, because the majority of
computing will simply assumed to
be done in the cloud. […]"

 Jack Uldrich, Futurist

Photo via http://www.prweb.com/releases/2011/12/prweb9052671.htm
© Microsoft

CONNECTING THE CLOUD WITH THE CROWD

"Everything will have moved into
the cloud: content, media, health
records, education.
Connecting the cloud with the
crowd will become a huge
business."

 Gerd Leonhard, Futurist

Photo via http://www.mediafuturist.com/about.html
© Microsoft

NEW ALGORITHMS AND TOOLS

"Predicting the future will be
common for the average person […]
New algorithms and tools will unlock
this rich source of data, creating
unprecedented insight. Cloud based
tools will allow anyone to mine this
data and perform what-if analysis,
even using it to predict the future."

 Dave Evans, Cisco Chief Futurist

Dave Evans, Cisco Chief Futurist
Photo via http://www.cisco.com/web/about/ac79/docs/bio/Dave_Evans_Exec_Bio_Final.pdf
© Microsoft

MY ANALYTICS PREDICTIONS FOR 2020

More + different data
More algorithms
More people
(everyone mines data)
More roles
(data scientists!)
More real-time
More social


General Chair MSR 2013 — Call for Papers
Thomas Zimmermann
Microsoft Research, USA International Working Conference on Mining Software Repositories
Program Co-chairs Sponsored by IEEE TCSE and ACM SIGSOFT
Massimiliano Di Penta May 18-10, 2013, San Francisco, CA, USA. Co-located with ICSE 2013.
University of Sannio, Italy http://msrconf.org twitter: @msrconf
Sunghun Kim
Hong Kong University of Science and Software repositories such as source control systems, archived communications between project
Technology, China personnel, and defect tracking systems are used to help manage the progress of software pro-
jects. Software practitioners and researchers are recognizing the benefits of mining this infor-
Chief of Data mation to support the maintenance of software systems, improve software design/reuse, and
Daniel Germán empirically validate novel ideas and techniques. Research is now proceeding to uncover the ways
University of Victoria, Canada in which mining these repositories can help to understand software development and software
evolution, to support predictions about software development, and to exploit this knowledge
Challenge Chair concretely in planning future development. The goal of this two-day working conference is to ad-
Alberto Bacchelli
vance the research and practice of software engineering through the analysis of data stored in
University of Lugano, Switzerland
software repositories.
Web Chair This year, MSR solicits three types of papers: research, practice, and data papers. As in previous
Julius Davies MSR editions, there will be a Mining Challenge and a special issue of best MSR papers in the Em-
University of British Columbia, Canada pirical Software Engineering journal.
Program Committee Important Dates
To be announced.
Research/practice papers: February 15, 2013 (abstracts: February 8)
Please see the conference website.
Data papers: March 4, 2013
Challenge papers: March 4, 2013

Call for Articles
SOFTWARE ANALYTICS: SO WHAT?
Special Issue of IEEE Software

Submission Deadline: 15 December 2012
Publication: July/August 2013

Software analytics are studies of software that lead to actionable changes to
projects. The feedback from analytics should alter decisions relating to the
business, management, design, development, or marketing of software
systems. These analytics can be applied to both the products of developers
(design documents, code, emails between team members) and to data generated by running programs (usage patterns,
economic effects of the running system). Often such analytics requires “big data” methods—visualizations or data
mining of large datasets.
In this special issue, we seek answers to seemingly simple questions: Do these software analytics really work? In
practice, what has actually been achieved? For a supposedly data-driven field, there are surprisingly few exemplar case
studies in the literature—of both successes and failures—in this area. Hence we have no answer for the business user
(or graduate student) who asks, “In this field, what are the best and worst practices, and why?”
The guest editors invite articles addressing the practical successes, as well as the practical drawbacks, of software
analytics. Such analytics includes the application of data mining tools to SE data (but can also include combinations of
automatic and manual data analysis). Topics for these submissions include but are not limited to the following:
 the added value of software analytics to the business community (if, indeed, it exists);
 the synergies (if any) that can be achieved by combining automatic and human insight about some industrial
problems;

smart analytics is Usage analytics
actionable Improving the Explorer for Windows 8
Debugging in the (very) large
Analytics for Xbox games
real time © Microsoft Corporation

sharing
diversity Development analytics
The SWEPT datamart
people Risk Assessment with CRANE
Branchmania – too many branches


Analytics for smarter software development

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Analytics for smarter software development

Similar to Analytics for smarter software development (20)

More from Thomas Zimmermann

More from Thomas Zimmermann (20)

Recently uploaded

Recently uploaded (20)

Analytics for smarter software development