I was recently asked to speak at ALM Days in Dusseldorf and more specifically to create a talk on Metrics and KPI’s for Quality. As I have been working a lot recently with evidence-based management. I am pretty sure that my session title translates as “Test management and reporting – KIP’s for better decisions” so I am going to concentrate on reporting and KPI’s as the session before mine is on Agile Testing. http://nkdalm.net/1fmvDrq
Martin HinshelwoodPrincipal at naked ALMSpecilise in ALM, TFS and ScrumHonor of being an ALM MVP And Professional Scrum Trainer with Scrum.orgIn recent years I have been increasingly interested in metrics but only in the last year have I seen metrics that are evidence-based leading metrics rather than circumstantial following metrics…. These metrics from a keen part of evidence-based management…Or: Metrics that matter for teams and organisations
Evidence, broadly construed, is anything presented in support of an assertion:Strongest type of evidence is that which provides direct proof of the validity of the assertion.Weakest type of evidence is that which is merely consistent with the assertion, but doesn’t rule out contradictory assertions, as in circumstantial evidence.
There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…
There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…
Team Foundation Server 2013 provides world class tool for reporting on the quality of your application. Come and see Martin show you which metrics are important and how to retrieve them from TFS. Learn how to use the built in reports and to create your own custom ones, all the while looking for those key quality measure that you can use to help identify smells in your process. Do you want to take advantage of Team Foundation Servers world class reporting to understand your Quality measures?
Circumstantial evidence is data that we use to support our analysis and with Team Foundation Server we get a plethora of options. We still need to make sure that we are not looking at the wrong metrics and at least TFS metrics are much more difficult to game as they come from an auditable system. However…
-So much data-Merill Lynch Team Lead data storyLeads to team metrics
A healthy Stories Overview report shows more progress on stories that appear near the top of the report. As the following illustration shows, the team has accomplished more work for those stories that appear first in the report. Stories are always listed according to their ranking. An unhealthy Stories Overview report shows one or more of the following indications:The team is making more progress on stories that have a lower rank than on stories that have a higher rank.More test are failing than are passing. Tests are failing for a story, but no bug work items are being created.
The X-axis lists the specific builds that the report includes, based on the filters that you have set for the platform, configuration, and build definition. Each vertical bar represents a set of data that was derived from one or more builds. In the code size variant of the report, each vertical bar’s length represents the size of the checked in code base. The bars are scaled so that the largest figure fits into the height of the chart. Manual tests can be run any time after the build, and they are associated with that build. Tests that have not been run yet are counted as "inconclusive."Active Bugs (count) - Line chart that depicts the number of bugs that were active at the time of the build. Code Churn (lines) - Line chart that depicts the number of lines of code that the team added, removed, and changed in the checkins before the build. The code churn is calculated by determining the number of lines of code that have been added, deleted, or modified to the build divided by the total lines in the build. Code Coverage (percent) - Line chart that depicts the percentage of code that the tests cover. Inconclusive Tests - Gray portion of the stacked bar chart, which indicates the number of tests that did not succeed or were paused. If the build did not succeed, the tests are either not counted or counted as inconclusive.Failed Tests - Red portion of the stacked bar chart, which indicates the number of tests that failed for the build.Passed Tests - Green portion of the stacked bar chart, which indicates the number of tests that passed for the build.
Less code coverage and more code churn - The following illustration shows a decrease in code coverage and an increase in code churn. This data is a clear warning that new code is being checked in without corresponding unit tests to cover it.Low rate of tests being run -The following illustration shows a low rate of tests being run. This data might indicate that the team is not performing enough testing. This blockage could indicate lack of resources, or testers might be doing something else, such as writing test automation instead of testing the current functionality. In either case, resource balancing might be warranted.High code churn, low rate of code coverage- High code churn suggests that bugs will be introduced as side effects of the changes. In a perfectly refactored project, you can see code churn with no change in code coverage or test pass rates. Otherwise, high code churn might indicate decreased coverage and the need to rewrite tests.
The chart shows a separate row for each combination of build definition, platform, and configuration. The report shows only those combinations that fall within the filters that you have specified for the report. At a glance, you can determine the success or failure of builds for the time period under review, as the following illustration shows.
The report lists the most recent builds first and contains build results that were captured during the specified time interval for all builds that were run, subject to the filters that you specified for the report.At a glance, you can determine the success or failure of several build definitions for the time period under review, as the following illustration shows.
(5 Minutes MAX)
(5 Minutes MAX) Show Excel connecting to the cube… don’t try to show data as you will not get any.+Show generating reports from Excel or Team Explorer
Leading to vanity metrics
-IAAI data storyLead to not the metrics you are looking for
Team Metrics don’t scale How do you create KPI’s for a devisionForresterResearched MetricsMetrics should match the goalIf the goal is to improve the organisation then use measures of value The Sandish Group50k Projects examinedInverse relationship between PBI Success and Value delivered
There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…+Net promoter score
Try something that is hot in the market; or,Use evidence based management:Observe and analyze how work is done now;Assess and analyze the capability of the people doing the work;Identify the most likely improvement to add value.Does not include marketing, sales, or new product ideas.
Try something that is hot in the market; or,Use evidence based management:Observe and analyze how work is done now;Assess and analyze the capability of the people doing the work;Identify the most likely improvement to add value.Does not include marketing, sales, or new product ideas
Scrum.org have come up with
You can use the QR code to access this presentation and all of the relevant links as well as additional information.Any questions?