For years, there have been stories of continuous delivery making teams awesome… but can CD make all teams awesome? And how? Dr. Nicole Forsgren will present data from over 20,000 technical professionals showing the central role that CD plays in software development and delivery. She will show you how doing CD can drive key organizational outcomes like profitability, productivity, and market share. Nicole also presents the key aspects of CD that make your DevOps awesome, like trunk-based development, test data, and test automation, and provides examples of success from teams undergoing their own technology transformations. The presentation also includes other important drivers of DevOps success, like lean product management and team culture. At the end of this talk, you will have the information to help you prove your case (to management or even yourself) about why CD and DevOps are essential to winning, as well as great stories and examples to really bring these concepts to life. You’ll leave with tips you can take back to get started on your own DevOps initiative.

1. Continuous Delivery:
Making DevOps Awesome
Nicole Forsgren, PhD
CEO and Chief Scientist, DevOps Research and Assessment (DORA)
Research Affiliate, Clemson University, Florida International University

2. “IT doesn’t matter.”
-- Nicholas Carr, 2003
@nicolefv

3. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

4. @nicolefv

5. @nicolefv

6. DevOps is
Technical practices
seen in Continuous Delivery,
Management practices
seen in Lean Management principles, and
Organizational Culture
@nicolefv
Research shows that these drive Organizational Performance
and IT Performance

7. DevOps is good for
organizations
@nicolefv

8. High Performing IT organizations
2x
More likely to exceed
Profitability,
Market share, and
Productivity goals
50%
Higher market cap
growth over 3 years*
The 2014 and 2015 State of DevOps Report
@nicolefv

9. Devops is good for IT
Measuring DevOps and IT Performance
- Deploy frequency (when business demands)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate
@nicolefv

10. High Performing DevOps teams
More agile
The 2016 State of DevOps Report
@nicolefv
200x
200x more frequent
deployments
2,555x
2,555x shorter lead
times

11. High Performing DevOps teams
More reliable
The 2016 State of DevOps Report
@nicolefv
24x
24x faster
recovery from failures
3x
3x lower
change fail rate

12. DevOps promises – and delivers
More throughput
More stability
In tandem. Without the tradeoffs that ITIL
calls for.
Let’s talk about what this means for us
@nicolefv

13. High Performing DevOps teams
More agile What does this mean for
200x
More frequent
deployments
The 2016 State of DevOps Report
New content delivery
Value/savings around A/B
testing
Value around speed to market
Compliance / regulatory
Security
2,555x
Faster lead times
@nicolefv

14. Evaluating well-designed and
executed experiments that were
designed to improve a key metric,
only about 1/3 were successful
at improving the key metric!
Online Experimentation at Microsoft, Kohavi et al http://stanford.io/130uW6X
@nicolefv

15. High Performing DevOps teams
More reliable What does this mean for
3x
Fewer deploy
failures
The 2016 State of DevOps Report
Value/savings around
reliability
Value/savings around uptime
Compliance
Security
Reputation around uptime,
compliance & security
24x
Faster MTTR
@nicolefv

16. @nicolefv

17. "Fixing it as soon as possible or
having compensating controls in
place days before could have
saved this entire breach from
occurring in the first place."

18. We know:
• IT Performance is comprised of throughput and
stability, and both are possible without tradeoffs
• IT Performance contributes to org performance
(profitability, productivity, market share)
So:
What drives IT and Organizational Performance?
@nicolefv

19. IT
Performance
Org
Performance???
@nicolefv

20. IT
Performance
Org
Performance???
1.
@nicolefv

21. IT
Performance
Org
Performance???
2. 1.
@nicolefv

22. IT
Performance
Org
Performance???
2. $$$1.
@nicolefv

23. IT
Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery

24. IT
Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery
• Effective test
data
management
• Test &
Deployment
automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security

25. IT
Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery
• Effective test
data
management
• Test &
Deployment
automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security

26. Test Data Management
Analysis showed that data
was creating bottlenecks in
the process.
Key activities in TDM
include:
• Test data masking and
subsetting
• Test data automation
• Integration with tooling
Outcomes:
• Faster time to market
• Improved quality and
customer experience
• Significant productivity
improvements and savings
• Improved testing
efficiencies
@nicolefv

27. Test & Deployment Automation
“By getting new releases and services out the door quickly, we can
provide a better experience to millions of car buyers and sellers and
continue to differentiate ourselves in a competitive market.”
-- Adam Mills, Senior Manager of Application Development,
AutoTrader.com
Using a DevOps approach to unify teams, they have been
able to automate their test, build, and deployment
Using these principles, Autotrader.com has been
able to:
• Reduce test, build, and deploy times
from weeks to just minutes
• Decrease software defects by 25%

28. IT
Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery also makes our work
feel better
• Effective test
data
management
• Deployment
automation
• Test automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security
Decreases
burnout and
deploy pain

29. Microsoft Engineering: DevOps Lessons
Thiago Almeida -- DevOps Days London, 2016
@nicolefv
Work/Life Scores
Before CD
38%
After CD:
75%
Source: https://vimeo.com/165184757

30. But what else drives IT
Performance?
The 2015 DevOps Survey of Practice and its resulting database are the property of Puppet Labs, Inc. and Gene Kim and Associates, LLC. All rights
reserved.
@nicolefv

31. IT
Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices

32. IT
Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer
feedback
• Small batches
• Making work
visible through
the value
stream

33. IT
Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer
feedback
• Small batches
• Making work
visible through
the value
stream

34. @nicolefv
Visual Displays
“If it moves, graph it.”
- Michael Rembetsy, Former VP Operations, Etsy

35. Making Work Visible Through the
Value Stream
“One of the greatest effects… is
that we are now asked much
earlier by the business to
participate in the requirements
gathering process.”
-- Project Manager at a.s.r.
By creating flowcharts to show work as it moved
throughout the value stream, a.s.r. was able to
leverage insights from all stakeholders including
business, test, and IT. Leveraging these tools,
teams could immediately understand changes
made to requirements and their impact.
Using this principle, a.s.r. was able to:
• Reduce test time preparation
time significantly
• Release software with no
incidents
• Deliver on time and
on budget

36. IT
Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices also make
work feel better
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer
feedback
• Small batches
and making
work visible Decreases
burnout

37. “I was trying to figure out why my team was working
themselves to death but not getting anything done… By
implementing WIP limits, we were able to focus on our
work. Finishing work feels better than sprinting and
feeling like a hero in the moment, because that’s only a
moment.”
- Julia Wester,
Development
Manager for Turner
Sports, Turner
Broadcasting
@nicolefv

38. But wait… there’s more

39. IT
Performance
Org
Performance
Continuous
Delivery &
Lean
Practices
@nicolefv
CD and Lean Management practices also
improve our culture and identity, which
improve IT & Org Performance
Westrum
Org Culture
Identity

40. Westrum Organizational Culture
@nicolefv
Pathological
Power-oriented
Bureaucratic
Rule-oriented
Generative
Performance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to
scapegoating
Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented

41. Identity
• I am glad I chose to work for this organization rather than
another company.
• I talk of this organization to my friends as a great company to
work for.
• I am willing to put in a great deal of effort beyond what is
normally expected to help my organization to be successful.
• I find that my values and my organization's values are very
similar.
• In general, the people employed by my organization are working
toward the same goal.
• I feel that my organization cares about me.
Adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and Kwok-Kee Wei (2005), “Contributing
Knowledge to Electronic Knowledge Repositories: An Empirical Investigation,“ MIS Quarterly,
29, 113-143.
@nicolefv

42. Google Team
Performance
Surprise! No magic
formula for what
makes the perfect
team
@nicolefv

43. Intuit
“By installing a rampant innovation culture, we
performed 165 experiments in the peak three months
of tax season.
Our business result? Conversion rate of the website is
up 50%. Employee result? Everyone loves it, because
their new ideas can make it to market. ”
- Scott Cook, Intuit founder
@nicolefv

44. Amazon
“I think building this culture is the key to innovation.
Creativity must flow from everywhere. Whether you
are a summer intern or the CTO, any good idea must
be able to seek an objective test, preferably a test that
exposes the idea to real customers. Everyone must be
able to experiment, learn, and iterate.”
- Greg Linden
@nicolefv

45. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

46. For more information:
@nicolefv
Sign up for our ROI
whitepaper & get peer-
reviewed research
devops-research.com

47. Thank you
nicole@devops-research.com
@nicolefv
nicolefv.com