3. a metric is a measure or a combination of measures for
quantitatively assessing, controlling or improving
a process, a product, a team
a KPI, Key Perfomance Indicator, is a (aggregate)
metric that:
is tied to a strategic objective;
have at least one defined time-bound target value
(number, range, limit, percentage, trend, variation)
Metric vs. KPI
5. actionable understandable
accessible
Many “recipes” and
characteristics, i.e.
SMART
Specific
Measurable
Achievable
Relevant
Timely
INVEST
Immediately actionable
Negotiable
Valuable
Estimable
Sized to fit
Testable
a good KPI
3 are enough
8. some examples (time, effort, scope)
Is planning accurate ?
How long does it take for a
requirement to be delivered
to customers ?
Are effort and cost estimates
accurate ?
How is effort split between
design, coding and testing ?
Are requirements satisfied ?
Are requirements changing ?
How often ?
How “big” is the software ?
Schedule Adherence &
Variance
Lead Time, Cycle Time
Slip Charts
Effort & Cost Adherence
Cost per Phase
Amount of Rework
Requirements adherence
Requirements volatility (churn)
Code size (KLOCs :( ,
#modules, #classes, …)
11. exponential metric growth: i.e. Defects
total # defects
# defects by category (i.e. critical, major, minor)
# non-functional defects (usability, performance)
# new defects / time
# defects fixed / time
# critical defects / time
# re-opened defects (regression)
# tests / defect
time required to fix a defect
# defects found in-house / total # defects (DRE)
# defects found / # test hours spent
etc.
15. Metrics should not scare or threaten people
Enforced metrics are often cheated or ignored
16. Cheating
copy & paste some code,
copy & paste unit tests making slight modifications
result: increased code coverage
write some buggy code and
quickly fix it
result: increased number of fixed bugs
(maybe you also get credit for additional LOCs!)
17. Sex, Lies & Statistics
(beware of wrong/biased numbers)
1920: “most criminals are farmers”
actually most people where farmers at that time
1940: “twins more likely if mothers are in their 25s-30s”
most mothers gave birth in their 25s-30s at that time
growth much easier when
starting from a small base
18. Code Quality Metrics
KLOCs
code reviews & specific analysis tools needed
how fragile you are
Complexity
Duplication
Smells
Churn
20. some Agile principles:
Create Value for the customer as early as
possible
Eliminate Waste (WIP, YAGNI)
Drive and Respond to Change, quickly
Time/Capacity Boxing (see Scrum and Kanban)
Provide Visibility into project progress
Enter Agile
21. productivity defects efficiency
effort customer satisfaction
velocity time requirements
size trend quality burndown roi
schedule business value capacity
complexity cumulative flow effectiveness
Googling with KPIs
22. Story Point
an arbitrary value to express effort, complexity and risk
associated to a user story, feature, task
Velocity
S
Velocity
# story points / interval (where
interval is usually a sprint or more
in general time)
Velocity is a team specific metric, it
cannot be used to compare
different teams
Velocity Improvement is a key goal
for Agile teams
24. -100
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14
added SP
SP carried over from
previous sprint
SP todo from initial list
ideal trend
velocity
added SP trend
actual trend
Release Burndown Chart
sprint
velocity
delay
new expected
completion date
SPs have to be dropped
to get back on track
25. 0
50
100
150
200
250
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
real SP left
real Xtra time
left
ideal burndown
ideal Xtra time
left
Sprint Burndown Chart
bad
baddays
delay
look, ma
instant feedback!
Xtra time = time reserved
for other tasks (emergencies,
bug fixing, reviews, etc.)
26. each story has one or more automated tests
when tests pass the story has been completed
=> you are forced to become agile:
cannot do BDUF*, must do automated testing/
continuous integration
must deliver early value
Compile
Build
Test
Deploy
Running Tested FeaturesRon Jeffries
C.I. Quality Feedback Loop*Big Design Up Front
27. Backlog
Story
Points
Value For
Customer
Value
Delivered
(*100)
story 1 13 21 162
story 2 21 13 62
story 3 34 21 62
story 4 5 8 160
story 5 5 5 100
story 6 13 3 23
story 7 8 8 100
story 8 3 1 33
story 9 13 5 38
story 10 21 5 24
Early Value Delivery (again)
Business Value Points
Desire of Customer
Value =
Cost of providing functionality
68.3% of value
already delivered
here
same velocity
greater value!
story points
28. Iteration Related Metrics
(%) Stories & Story Points Completed
(%) Stories added/removed (Sprint & Release)
Stories unfinished/moved to Next Sprint
Sprints Moved to Next Release
Lead & Cycle Time (Stories & Defects)
Average Age of Stories and Defects
(%) Failed Builds, (%) Failed Tests
Defects Added & Fixed (absolute & trend)
29. WIP
Cumulative Flow Diagram
to do
doing
done
cycle time
cycle time
WIP
WIP is
increasing
done
doing
to do
Work In Process (WIP) and Cycle Time
should be minimized
30. Some Kanban Specific Metrics
Cycle Time =
Number of Things in Process/
Average Completion Rate
Little’s Law
time spent in each lane ?
bottlenecks ?
cycle time
lead time
Flow = Speed * Density,
Density Speed
=> Traffic Jam
40 60 25
ouch!
ouch!
31. first thing in the morning: dashboard!
Short Term vs. Long Term Dashboard.
Short Term: build ok, automated tests ok, value earned, schedule
(burndown), critical bugs entered, critical bugs still there, impediments, ...
Long Term: trends: failed builds, failed tests, # of tests, velocity, earned
value per month, # and type of bugs, …
i.e. increase test coverage
32. Essential Tools
a big board and lots of cards (hard to extract metrics though)
source code management (obvious)
automated build & test suites (CI)
issue tracking*
some/many of these tools may be
part of integrated product suites
*unless you belong to a small elite
code analysis
agile management system
dashboard