“Every line is the perfect length if you don't measure it.” - Marty Rubin So your organization has embarked upon a transformation to be more nimble and responsive by employing the latest tools and thinking in the Agile and DevOps arena. In this transformational context, how do you know that your initiatives are effective? Empirical measurements should provide insights on business value flow and delivery efficiency, allowing teams and organizations to see how they are progressing toward achieving their goals, but all too often we find ourselves mired in measurement traps that don't quite provide the right guidance in steering our efforts. Rooted in contemporary thinking and tested in practice, this talk explores the principles of good measurement, what to measure, what not to measure, and enumerates some key metrics to help guide and inform our Agile and DevOps efforts. If done right, metrics can present a true picture of performance, and any progression, digression of these metrics can drive learning and improvement. It is our hope that this session inspires organizations and teams to start or take a fresh look at implementing a valuable measurement program.

1. What’s Measured Improves
Metrics
That
Matter
!
Raj Indugula
raj.indugula@lithespeed.com
Rob Brown
robert.david.brown@gmail.com

2. Agenda
• Why and What of Metrics
• Principles of Measurement
• Key Drivers & Metrics
• Key Takeaways
• Q&A

3. What does
mean in today’s
Lean-Agile-DevOps
world?
3

4. What is Measurement?
Measures
• I have 5
apples
• # incidents
Metrics
• I have 5 more
apples than
yesterday
• % of Sev1
incidents since
feature rollout
Observation that reduces uncertainty…
4

5. Are Metrics Evil?
5

6. 6

7. What Makes A Good Metric?
http://www.flickr.com/photos/circasassy/7858155676/
Understandable
Comparative
Behavior changing
7

8. Measure
movement
towards your
business goals &
outcomes
Where are you going?
8
Picture source New Relic

9. KEY DRIVERS & METRICS
Development ReleaseQA OperateDeploy
Customer
Success
Business
Success

10. 1. Business Success
• Starts with Measurable Objectives
• Indicators for success
• Market share
• New business service enablement
10

11. #ofStories
0
40
80
120
160
Sprints
78 (6/8-6/21) 80 (7/6-7/19) 82 (8/3-8/16) 84 (8/31-9/13) 86 (9/28-10/11) 88 (10/26-11/8)
Scope (Total Stories Planned) Total Completed (Cumulative) Stories Completed Per Sprint
Projected (Optimistic) Projected (Pessimistic) Projected (Median)
26
19
483
2019
1116
108106
100
81
77
6966
46
27
16
138138138138
134
128
121118
109
99
94
88
16
27
46
66 69
77
81
100
106 108
88
94
99
109
118 121
128
134
138 138 138 138
Empirical Data Drives Enquiry and Adjustment
11Business success
Q: When are all planned stories estimated to be completed?
A: End of Sprint 90 (12/06/16)
Q: What is the total number of stories estimated to be
completed by 11/08/2016?
A: 119 Stories based on average velocity
Q: Can we deliver all current planned stories by 10/16/2016?
A: Not Likely, given the current trend
Fixed Scope
Fixed Date
Fixed Scope
& Date

12. Business Success
• Average Revenue per User
• Customer Lifetime Value
• Daily/Monthly Active Users
• Average Session Time
• % Change in Customer Volume
• Feature Metrics
Business
Success

13. 2. Customer/User Success
• Frequency of key
transactions
• Amount of time spent
in the application
• User satisfaction
survey results
• A/B test results
• Customer ticket
volume
www.glasbergen.com
13

14. Product 1
0
23
45
68
90
7/26/2020
8/2/20208/9/2020
8/16/2020
8/23/2020
8/30/2020
9/6/2020
9/13/2020
9/20/2020
9/27/2020
10/4/2020
10/11/2020
4
11 10
6
11
4 4
8
1
14
6
0
90
79
69
63
52
48
44
36 35
21
15 15
Pending
Resolved
Product 2
0
75
150
225
300
7/26/2020
8/2/20208/9/2020
8/16/2020
8/23/2020
8/30/2020
9/6/2020
9/13/2020
9/20/2020
9/27/2020
10/4/2020
10/11/2020
18
58
9 2 6 10
20 25 28
19 22
3
229
171
162 160 154
144
124
99
71
52
30 27
Pending
Resolved
Customer Satisfaction
“You Build it, You Own It”
Improving Responsiveness to Issues

15. Customer Success
Customer
Success
• Customer Ticket Volume
• Net Promoter Score
• Net Value Score
• Customer Anxiety
• Reliability

16. 3. Operations Success
• Availability
• Performance
• Efficiency
• Support
16
http://landing.google.com/sre/book/
Deploy Release Operate

17. Improving Deployment Success Rate, Not Just Frequency
17Operations success
Build Daily Success Rate
Date
Build name
9/12/2016 9/13/2016 9/14/2016 9/15/2016 9/16/2016
Test Suite 1
Test Suite 2
Test Suite 3

18. Operate
Deploy Release Operate
• Deployment Lead Time
• MTTR
• % of Failed
Deployments
• Deployment frequency,
duration
• Release
Frequency
• Time/Cost per
Release
• Predictability
• MTTR
• Performance -
Response Time
• Availability
• Security Pass Rate

19. 4. Development Success
• Lead time for changes
(from development to
deployment)
• Deployment frequency
• % Failed deployments
• Incident severity
• Outstanding defects
Develop QA Deploy

20. 0%
25%
50%
75%
100%
Sprint83
Sprint84
Sprint85
Sprint86
Other
New Capability
Major Interface
Product Enhancements
Production Incidents
Technical Debt
Arch. Improvement
Insight into Team Capacity and Work Items
20Development success
Balancing New Feature Work with Maintenance and Improvement

21. Development
• Development Lead
Time
• Idle Time
• Cycle Time
• Work in Progress-
Technical Debt
• Rework
• Idle Time
• MTTD
• Defects
• Deployment Lead Time
• MTTR
• % of Failed
Deployments
• Deployment frequency,
duration
DeployQADevelopment

22. Outcome Title: Provide a common
platform for CI/CD - PaaS/CaaS
Outcome Description: Provide
teams a ubiquitous, self service
platform for build, deploy and test.
Measure: Team Adoption rate
Baseline: 1 team/month
Budget: 6 million
Dependancies: platform & security
engineers, IaaS provider
Adoption Rate Measures:
Features
Frequency
Form Factor
New Service
USER.id Date Event Form Count
billy.bob 2/10/17 login IDE 3
suzy.q 2/11/17 image stream GUI 4

23. • Average Revenue
per User
• Customer Lifetime
Value
• Daily/Monthly
Active Users
• Average Session
Time
• % Change in
Customer Volume
• Feature Metrics
• Development
Lead Time
• Idle Time
• Cycle Time
• Work in
Progress-
Technical Debt
• Rework
• Idle Time
• MTTD
• Defects
• Deployment
Lead Time
• MTTR
• % of Failed
Deployments
• Deployment
frequency,
duration
• MTTR
• Performance
- Response
Time
• Availability
• Security Pass
Rate
• Customer Ticket
Volume
• Net Promoter
Score
• Net Value Score
• Customer Anxiety
• Reliability
• Release
Frequency
• Time/Cost
per Release
• Predictability
Accelerate Value
Delivery
Balance speed, risk, quality &
cost
Reduce time to obtain/respond
to customer
Now, All Together
23
Operations SuccessDevelopment Success

24. Monitoring Framework
• Aggregate metrics from
all success contexts
• See the entire system
• Enables visualization,
anomaly detection,
trending, alerting…
Source Adaptation: Turnbull, The Art of Monitoring, Kindle edition, chap. 2.
Business
Success
Customer
Success
Development
Success
Operations
Success
24
Event Aggregator

25. https://github.com/capitalone/Hygieiahttps://github.com/capitalone/Hygieia

26. KEY TAKEAWAYS

27. • Keep inventory of metrics small
• Measure outcomes, not individuals
• Monitor balanced set of metrics
• Monitor trends, not data points
• Share Widely – Interaction encourages
exploration
“If you measure me in an illogical way…don’t
complain about illogical behavior” - Goldratt
27

28. Next Steps
Clear Objective Well formed
metrics
Measure &
Improve
“Metrics illuminate, not indicate”
– George Dinwiddie

29. REFERENCES

30. • http://onstartups.com/tabid/3339/bid/96738/Measuring-What-
Matters-How-To-Pick-A-Good-Metric.aspx
• http://www.slideshare.net/GoAtlassian/understanding-metrics-
what-to-measure-and-why-john-custy
• http://www.slideshare.net/TeamQualityPro/the-good-the-bad-
and-the-metrics
• http://devopsenterprise.io/media/
DOES_forum_metrics_102015.pdf
• http://devops.com/2014/11/10/devops-scorecard/
• http://www.datical.com/blog/9-metrics-devops-teams-tracking/
• http://devops.com/2015/01/26/metrics-devops/
• https://blog.appdynamics.com/devops/quantified-devops/
• http://www.slideshare.net/jedi4ever/devops-metrics
• http://www.slideshare.net/ITRevolution/does15-troy-magennis-
and-julia-wester-metrics-and-modeling-helping-teams-see-how-
to-improve
• https://thenewstack.io/tag/ebook5-monitoring-and-
management/
• https://www.artofmonitoring.com/

31. Pin Your Metric
UsefulUseless