KPIs für schnellere Entscheidungen mit Testmanagement und Reporting
•
7 likes•2,382 views
I was recently asked to speak at ALM Days in Dusseldorf and more specifically to create a talk on Metrics and KPI’s for Quality. As I have been working a lot recently with evidence-based management. I am pretty sure that my session title translates as “Test management and reporting – KIP’s for better decisions” so I am going to concentrate on reporting and KPI’s as the session before mine is on Agile Testing. http://nkdalm.net/1fmvDrq
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to KPIs für schnellere Entscheidungen mit Testmanagement und Reporting
Similar to KPIs für schnellere Entscheidungen mit Testmanagement und Reporting (20)
More from Martin Hinshelwood
More from Martin Hinshelwood (8)
Recently uploaded
Recently uploaded (20)
KPIs für schnellere Entscheidungen mit Testmanagement und Reporting
- 1. KPIs für schnellere Entscheidungen Martin Hinshelwood | naked ALM Consulting Consultant | MVP | Ranger | Trainer | Coach martin@nakedalm.com | http://nakedalm.com/blog
- 2. Agenda Data, data, everywhere Creating your own reports Beyond the team, metrics that really matter © naked ALM – Martin Hinshelwood 2013
- 3. Data, data, everywhere © naked ALM – Martin Hinshelwood 2013
- 4. Visual Studio ALM Data Overload Env-01 Env-02 Env-03 Env-04 PROD © naked ALM – Martin Hinshelwood 2013
- 5. Mmmm data, data, everywhere © naked ALM – Martin Hinshelwood 2013
- 6. Stories Overview © naked ALM – Martin Hinshelwood 2013
- 7. Build Quality Indicators © naked ALM – Martin Hinshelwood 2013
- 8. Build Success Over Time © naked ALM – Martin Hinshelwood 2013
- 10. Interpreting Reports Remember its not the values that matter… …but the trend of those values over time © naked ALM – Martin Hinshelwood 2013
- 11. Creating custom reports © naked ALM – Martin Hinshelwood 2013
- 12. © naked ALM – Martin Hinshelwood 2013
- 14. Here are some I prepared earlier © naked ALM – Martin Hinshelwood 2013
- 15. Avoid: Vanity Metrics PROJECT ESTIMATE OVERVIEW Actual (h) 4321 4321 6019 5234 PROJECT A PROJECT B PROJECT C PROJECT D 1296 1234 2502 2034 5234 7118 Original Estimate (h) PROJECT E © naked ALM – Martin Hinshelwood 2013
- 16. Team Metrics Only If you are looking for metrics to compare teams If you are looking for metrics to roll up If you are looking for metrics to measure your organisational goals Then… © naked ALM – Martin Hinshelwood 2013
- 17. Beyond the team, metrics that really matter © naked ALM – Martin Hinshelwood 2013
- 18. Organisational Metrics Revenue per Employee Release Frequency Product Cost Ratio Employee Satisfaction Release Stabilization Installed Version Index Customer Satisfaction Cycle Time Usage Index Innovation Rate Defects Business outcomes (Value) Time to Market (Lead time) Ability to Innovate (Quality) © naked ALM – Martin Hinshelwood 2013
- 19. Direct Evidence Is Consolidated In One Indicator Business outcomes (Value) Time to Market (Lead time) Ability to Innovate (Quality) Agility Index © naked ALM – Martin Hinshelwood 2013
- 20. Relating Outcome To Value Creation Direct Evidence Circumstantial Evidence Diagnose Time to Market (Lead time) Business outcomes (Value) Ability to Innovate (Quality) • Skills, Knowledge, Understanding Product managers Managers Developers • Practices, Tools, Architectures, Standards Process Agility Index Measure Improve • Scrum • Self-organization • Product Backlog • Sprint Planning • Daily Scrum • Sprint Review • Sprint Retrospective • Scaling Scrum Productivity • Definition of Done • Testing • Clean code • Test-Driven development • Continuous Integration • Emergent Architecture Value • Accountability • Transparency • Product Backlog • Alignment • Release planning and orientation • Portfolio Management Quality • Engineering standards • Architecture • QA • ALM Enterprise • Communication • Organization • Culture • People Practices • Sales • Lean © naked ALM – Martin Hinshelwood 2013
- 21. Practices Process • • • • • • • Scrum Self-organization Product Backlog Sprint Planning Daily Scrum Sprint Review Sprint Retrospective • Scaling Scrum Productivity • Definition of Done • Testing • Clean code • Test-Driven development • Continuous Integration • Emergent Architecture Value • • • • • Accountability Transparency Product Backlog Alignment Release planning and orientation • Portfolio Management Quality • Engineering standards • Architecture • QA • ALM Enterprise • • • • • • Communication Organization Culture People Practices Sales Lean © naked ALM – Martin Hinshelwood 2013
- 22. Evidence of Capabilities Organizational Patterns Knowledge Trends Evaluated through observation of practices Evaluated through assessments 100 90 80 70 60 50 40 30 20 10 0 Developer Lead Developer Project Management Por olio Management Product Manager Capability Manage © naked ALM – Martin Hinshelwood 2013
- 23. Measure Practice Patterns © naked ALM – Martin Hinshelwood 2013
- 24. Evidence through Agility Index Snapshot Measures an organizations value across 3 dimensions and aggregated into one number, the Agility Index Weighted Index to track trends and ROI © naked ALM – Martin Hinshelwood 2013
- 25. Adapting patterns while controlling risk with Guided Improvement Delivers change in short cycles Inspects evidence of change © naked ALM – Martin Hinshelwood 2013
- 26. References http://nkdalm.net/AgilityIndex http://nkdalm.net/RadicalMgt © naked ALM – Martin Hinshelwood 2013
- 27. naked ALM Consulting Every company deserves working, high quality software on a regular cadence that successfully and consistently meets their customers needs http://nkdalm.net/1kFukvN Questions? Testmanagement und Recording KPIs für schnellere Entscheidungen Martin Hinshelwood | naked ALM Consulting Consultant | MVP | Ranger | Trainer | Coach martin@nakedalm.com | http://nakedalm.com/blog © naked ALM – Martin Hinshelwood 2013
Editor's Notes
- Martin HinshelwoodPrincipal at naked ALMSpecilise in ALM, TFS and ScrumHonor of being an ALM MVP And Professional Scrum Trainer with Scrum.orgIn recent years I have been increasingly interested in metrics but only in the last year have I seen metrics that are evidence-based leading metrics rather than circumstantial following metrics…. These metrics from a keen part of evidence-based management…Or: Metrics that matter for teams and organisations
- Evidence, broadly construed, is anything presented in support of an assertion:Strongest type of evidence is that which provides direct proof of the validity of the assertion.Weakest type of evidence is that which is merely consistent with the assertion, but doesn’t rule out contradictory assertions, as in circumstantial evidence.
- There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…
- There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…
- Team Foundation Server 2013 provides world class tool for reporting on the quality of your application. Come and see Martin show you which metrics are important and how to retrieve them from TFS. Learn how to use the built in reports and to create your own custom ones, all the while looking for those key quality measure that you can use to help identify smells in your process. Do you want to take advantage of Team Foundation Servers world class reporting to understand your Quality measures?
- Circumstantial evidence is data that we use to support our analysis and with Team Foundation Server we get a plethora of options. We still need to make sure that we are not looking at the wrong metrics and at least TFS metrics are much more difficult to game as they come from an auditable system. However…
- -So much data-Merill Lynch Team Lead data storyLeads to team metrics
- A healthy Stories Overview report shows more progress on stories that appear near the top of the report. As the following illustration shows, the team has accomplished more work for those stories that appear first in the report. Stories are always listed according to their ranking. An unhealthy Stories Overview report shows one or more of the following indications:The team is making more progress on stories that have a lower rank than on stories that have a higher rank.More test are failing than are passing. Tests are failing for a story, but no bug work items are being created.
- The X-axis lists the specific builds that the report includes, based on the filters that you have set for the platform, configuration, and build definition. Each vertical bar represents a set of data that was derived from one or more builds. In the code size variant of the report, each vertical bar’s length represents the size of the checked in code base. The bars are scaled so that the largest figure fits into the height of the chart. Manual tests can be run any time after the build, and they are associated with that build. Tests that have not been run yet are counted as "inconclusive."Active Bugs (count) - Line chart that depicts the number of bugs that were active at the time of the build. Code Churn (lines) - Line chart that depicts the number of lines of code that the team added, removed, and changed in the checkins before the build. The code churn is calculated by determining the number of lines of code that have been added, deleted, or modified to the build divided by the total lines in the build. Code Coverage (percent) - Line chart that depicts the percentage of code that the tests cover. Inconclusive Tests - Gray portion of the stacked bar chart, which indicates the number of tests that did not succeed or were paused. If the build did not succeed, the tests are either not counted or counted as inconclusive.Failed Tests - Red portion of the stacked bar chart, which indicates the number of tests that failed for the build.Passed Tests - Green portion of the stacked bar chart, which indicates the number of tests that passed for the build.
- Less code coverage and more code churn - The following illustration shows a decrease in code coverage and an increase in code churn. This data is a clear warning that new code is being checked in without corresponding unit tests to cover it.Low rate of tests being run -The following illustration shows a low rate of tests being run. This data might indicate that the team is not performing enough testing. This blockage could indicate lack of resources, or testers might be doing something else, such as writing test automation instead of testing the current functionality. In either case, resource balancing might be warranted.High code churn, low rate of code coverage- High code churn suggests that bugs will be introduced as side effects of the changes. In a perfectly refactored project, you can see code churn with no change in code coverage or test pass rates. Otherwise, high code churn might indicate decreased coverage and the need to rewrite tests.
- The chart shows a separate row for each combination of build definition, platform, and configuration. The report shows only those combinations that fall within the filters that you have specified for the report. At a glance, you can determine the success or failure of builds for the time period under review, as the following illustration shows.
- The report lists the most recent builds first and contains build results that were captured during the specified time interval for all builds that were run, subject to the filters that you specified for the report.At a glance, you can determine the success or failure of several build definitions for the time period under review, as the following illustration shows.
- (5 Minutes MAX)
- (5 Minutes MAX) Show Excel connecting to the cube… don’t try to show data as you will not get any.+Show generating reports from Excel or Team Explorer
- Leading to vanity metrics
- -IAAI data storyLead to not the metrics you are looking for
- Team Metrics don’t scale How do you create KPI’s for a devisionForresterResearched MetricsMetrics should match the goalIf the goal is to improve the organisation then use measures of value The Sandish Group50k Projects examinedInverse relationship between PBI Success and Value delivered
- There are few metrics that follow the evidence-based approaches described in “The Leaders Guide to Radical Management by Steven Denning” but one such emerging dataset is, the Agility Index from Scrum.org. It looks at a number of areas…+Net promoter score
- Try something that is hot in the market; or,Use evidence based management:Observe and analyze how work is done now;Assess and analyze the capability of the people doing the work;Identify the most likely improvement to add value.Does not include marketing, sales, or new product ideas.
- Try something that is hot in the market; or,Use evidence based management:Observe and analyze how work is done now;Assess and analyze the capability of the people doing the work;Identify the most likely improvement to add value.Does not include marketing, sales, or new product ideas
- Scrum.org have come up with
- You can use the QR code to access this presentation and all of the relevant links as well as additional information.Any questions?