DevOps Metrics - Lies, Damned Lies and Statistics

•

19 likes•6,723 views

Gaetano Mazzanti

my presentation at DevOps Day 2015 Italy

Data & Analytics

DevOps Metrics
lies, damned lies and statistics
Gaetano Mazzanti
@mgaewsj
agile42

why do we need metrics?
key reason: to improve

metrics & statistics still require
reasoning and visual examination
beware

same mean, variance & correlation
(7.50 3.75 0.816)

use your eyes
(same mean, variance & correlation)
median = 7.58
median = 8.14
median = 7.11
median = 7.04

ecological fallacy
average math score
50/100 60/100
group A
70% of people in A have
better score than B!
group B
70%

ecological fallacy
50/100 60/100
70%
Group A Group B
# score # score
50 45 70 43
50 55 30 100
group A group B
average math score

Simpson’s paradox
Global Natural Treat
Live 108 153
Die 123 120
Natural 47% live
Treat 56% live
Women Natural Treat
Live 57 32
Die 100 57
Natural 36,3% live
Treat 36,0% live
Men Natural Treat
Live 51 121
Die 23 63
Natural 69% live
Treat 45% live

deployment frequency
lead time for changes
mean time to recover
change fail rate
how IT performance
was measured
!?!?!?

ITIL KPIs
“ITIL Key Performance Indicators (ITIL KPIs)
are used to assess if the processes
of an IT organization are running
according to expectations”
and if not…
just kidding

it’s easy to get lost
in a maze of (not relevant) data

what do you want
to learn?
key question about your metrics

improvement loops
build/
change
measurelearn
experiment
actionable
metric
hypothesis

which metrics
matter to customers
no yes
end-to-end
(global)
functional
(local)
typical
ideal

service oriented mindset
DevOps as a service provided
to deliver value to the business

pizza delivery
fast delivery
accuracy and quality
predictability
what matters
to customers

let work ﬂow
ﬂow is the movement and delivery
of customer value through a process

derive from poor ﬂow
slow delivery
low quality
unpredictability

just 3 metrics?
Work In Progress
Lead Time
Throughput

Little’s Law
Items In Queue = Arrival Rate * Waiting Time
Lead Time = Work In Progress / Throughput

0
1
2
3
4
5
6
7
8
1-Feb 3-Feb 5-Feb 7-Feb 9-Feb 11-Feb 13-Feb 15-Feb 17-Feb 19-Feb 21-Feb
scatterplot
54%
71%
88%
96%
lead time
(days) average

scatterplot
source ActionableMetrics book
Lead

scatterplot (only bugs)
source ActionableMetrics book
Lead

frequency distribution
source ActionableMetrics book
Lead

___
efﬁciency
process efﬁciency =
total time
active time________
20
5
= 25%
!1!!2!!3!!4!!5!!6!!7!!8!!9!10!11!12!13!14!15!16!17!18!19!20!
elaborate do validate deliver
waiting
active

SLAs
Service Level Agreements
agreementexpectation
a SLA is a contract
between a service provider and the user/
customer that deﬁnes the level of service
expected from the service provider
i.e. we expect an item to ﬂow through the process and exit
in 5 days or less with an 85% probability of success

SLAs – some hints
do not set a SLA without analyzing Lead Time data
do not allow a SLA to be set by someone external to
your group
do not set a SLA without collaborating with
customers and/or other stakeholders
use different SLAs for different Work Item Types

slack – avoid full utilization
absorb variations
% capacity
utilization
queuesize
queue size
grows
exponentially
at high capacity0
5
10
15
20
25
0 10 20 30 40 50 60 70 80 90 100

your policies shape your data
your data shape your policies

a possible approach
understand sources of dissatisfaction
analyze demand & capacity
discover work item types
measure ﬂow
set SLAs
setup metric based improvement experiments
(similar to STATIK…)

obsessions you should have
improve process continuously
remove problems/impediments asap
get help from metrics

Gaetano Mazzanti
@mgaewsj
gaetano.mazzanti@agile42.com
agile42

What's hot

The disconnect between the delivery organization and the business is prevalent in the software industry. Somewhere along the line, the real vision behind our projects gets lost. We all know it. Can better metrics help? This session examines some common and not-so-common metrics and introduces Evidence Based Management as a guide for continuously measuring your business goals, aligning them with your software development efforts, and then deciding what to do next.

Agile Metrics, Value, and Softwre

Don McGreal

It’s been said: If you can’t measure it, you can’t improve it. For most agile teams burndown charts and some type of velocity measurement are all they are doing. However, with just a few more metrics, you can gain substantial insight into how teams are performing and identify improvement opportunities. Andrew Graves explores seven key metrics―Effort by Class of Service, Accuracy of Estimation, Cost per Point, and four others―to measure how your team is doing and make adjustments in real time. Andrew illustrates how to use these metrics to communicate progress to stakeholders. Discover how to use these metrics to identify and analyze trends that lead to performance improvement ideas and strategies. Learn how to use these seven metrics to monitor the impact of changes made to verify they are bringing the hoped-for difference.

Seven Key Metrics to Improve Agile Performance

TechWell

Metrics - You are what you measure (DevOps Perth)

Rob Crowley

The Service-Delivery Review: The Missing Agile Feedback Loop

Matthew Philip

by Dave Bales This presentation aims to raise awareness that default Product Ownership (that uses only ordering and prioritising PBI’s) is not enough to be successful. A higher level of consciousness is required to appreciate the challenges and dynamics of the work at hand. Devising evolutionary strategies to cater for the undercurrents, changing landscape and the opposing dynamics in the product and its backlog, may help to tip the balance in favour of being more successful. Project dynamics can include: Releasing a new insurance app to millenials on a smartphone platform. The challenge with this is that more customer testing and an exploratory approach of how milenials think will be required, hence each sprint goal may include working toward a customer testing objective; A compliance risk project where compliance objectives are static. How those objectives are satisfied is variable, hence an evolving strategy to use an iterative approach is to be considered. The first iteration would be to neutralise the compliance risk, starting with a manual process. Successive iterations would then be run by automating and adding more value; An integration risk project, where a central framework has to integrate with several other systems. An evolving strategy would focus on setting sprint goals to achieve integration, as well as regression testing objectives to neutralise the integration risk early

Art of Product Ownership vs. The Conveyor Belt Approach

Scrum Australia Pty Ltd

Agile Metrics

Mikalai Alimenkou

To Estimate or Not to Estimate, Is that the Question? (2017 Better Software C...

Matthew Philip

Flow Efficiency - The most powerful improvement driver - Lean Kanban Central ...

Marcio Sete

I gave this presentation at Lean Kanban Asia-Pacific conference in Bangalore, India on December 11th, 2014 and at AgileDC on Washington, USA on October 21st, 2014. I have several recent blog posts on this topic, This search link should get most of them: http://connected-knowledge.com/?s=lead+time. If you need one "best" post, here it is, Inside a Lead Time Distribution: http://connected-knowledge.com/2014/09/07/inside-lead-time-distribution/

Lead Time: What We Know About It...

azheglov

Slides as presented on Dec 8, 2020 at FlowConf organized by Agile Kanban Istanbul. https://www.flowconf.com/ Organizational change often stalls out at departmental boundaries, whether that is IT or another division. How do we help organizations connect vertically and horizontally to realize the outcomes that they have when undertaking large-scale change efforts? Join this session to learn from a case study of a bank that combined flight levels and metrics to bridge their departmental boundaries and recognize gains not only in software delivery effectiveness but unifying higher-level strategy.

Metrics at Every (Flight) Level [2020 Agile Kanban Istanbul FlowConf]

Matthew Philip

Agile Workshop: Agile Metrics

Siddhi

Metrics As A Learn And Change Agent

Gaetano Mazzanti

These are the slides for the webinar https://vimeo.com/manage/296007941 A lack of clarity costs companies, educational institutions, government agencies, and nonprofits billions of dollars a year. Beyond the red ink, this lack of clarity also inserts unnecessary risk, demotivates team members, and causes customers to question whether the organization can deliver value. On a personal front, a lack of clarity creates interpersonal tension and hurts the mission of otherwise well-meaning leaders. In this webinar, Karen shares what clarity is and how a lack of clarity leads to poor performance--both organizationally and personally--and eroding trust. She shares powerful ways to operate with greater clarity to unleash the potential of people and organizations alike. To supplement the webinar, consider taking our free quiz to assess how you and your organization currently rate (www.clarityfirstquiz.com) or purchase the book (www.clarityfirstbook.com).

Clarity First: What it is. Why you need it. How to get it.

TKMG, Inc.

Improving Agility (Learning from Maersk Line's Journey) | Özlem Yüce | Agile ...

Agile Greece

Recorded webinar: http://slidesha.re/1juuPs4 To subscribe: http://ksmartin.com/subscribe To purchase the book: http://bit.ly/VSMbk Value Stream Mapping is a powerful way to improve performance across a large portion of an enterprise. But, before you dive into mapping, there are significant steps you must take to assure a successful outcome. In this webinar, award-winning author, Karen Martin shares the proper preparation an organization must take to reap the full set of benefits Value Stream Mapping offers. Topics include: Leadership – what is their role? Scoping – how "big” should you go? Team formation – who are the right people to include? Facilitator selection – what traits and skills are needed? Charter development and socialization – it’s far more than a planning tool! Logistics & communication – how do you make sure everyone is engaged and prepared? Watch this webinar for a no-nonsense discussion about the key success factors and common failings in preparing to value stream map.

Value Stream Mapping: What to Do Before You Dive In

TKMG, Inc.

LKRU14 and LKAPAC: Five Numbers

azheglov

Presented by Roy Vasher, WHIN Education Consultant, Purdue University Steve Dunlop, Managing Director of Dauch Center for DCMME & GSCMI Ananth Iyer, Senior Associate Dean, Purdue University Dr. Angus McLeod, VP of Coaching at AMA Hosted and moderated by Mark Graban This webinar is focused on manufacturing, but will have important lessons related to leadership and supply chain operations -- important issues in any industry these days. Listen to a quick preview: In this webinar, you'll learn: How focusing on Technology, Product, People and Process (TP3)are key to holistic management of pandemics through to the New Normal The role of Agility in keeping ahead of events Why and how communication can create trust or lose trust A psychological and work-culture ethos for opportunity, not threat Why now is the time to shift focus from fire-fighting to strategy How Infection control audit and measures leverage Value Stream Mapping (VSM) to reduce infection threats and helps us focus on efficacy enhancement, not just costs/wastes Why investing in technology is immediately a more affordable and strategic advantage then it was before the pandemic Why our multi-tasking tool helps during the pandemic and helps beyond the New Normal Why re-shoring strategies now make even more sense than they did prior to the pandemic

A TP3 Approach to Manage Manufacturing Competitiveness After the Pandemic

KaiNexus

This presentation strives to shift common perceptions around estimating, encouraging more productive, strategic practices that improve project outcomes for freelancers and agencies alike. Building on this philosophical shift, you will learn how to produce more reliable estimates based on granular tasks; how to use simple yet powerful estimation techniques such as the PERT method; and how to gather the information needed to produce an accurate estimate. Finally, based on 4 years of experience managing the 10up production team, I’ll share best practices, tips, and tools for successfully communicating estimates internally and externally as well as how to leverage estimates to inform agency resource planning and growth.

The Importance of Estimating - WCMKE 2015

Jess Jurick

Top Metrics for Agile @Agile NCR2011

Priyank Pathak

Recorded webinar: http://slidesha.re/1sKe9P7 Subscribe: http://ksmartin.com/subscribe Purchase the book: http://bit.ly/VSMbk Too many organizations today suffer from silo-centric behavior and intra-organizational conflict. Yet most don’t understand what’s holding them back from achieving outstanding performance. Value Stream Mapping (VSM), popularized in Mike Rother and John Shook’s landmark book Learning to See, is a highly effective method for improving performance across an organization. However, VSM is commonly misunderstood, misapplied and under-utilized. Properly executed, value stream mapping is much more than an operations design tool. It also results in a more engaged and knowledgeable leadership team; surfaces hidden cultural and organizational issues; and aligns values, resources, goals and priorities. In this webinar, you’ll learn how to * Use value stream mapping as an organizational transformation tool * Properly scope and plan for a value stream mapping activity * Obtain leadership consensus about the future state design * Create and execute a highly effective Value Stream Transformation Plan

Value Stream Mapping: From Tool to Management Practice

TKMG, Inc.

What's hot (20)

Agile Metrics, Value, and Softwre

Seven Key Metrics to Improve Agile Performance

Metrics - You are what you measure (DevOps Perth)

The Service-Delivery Review: The Missing Agile Feedback Loop

Art of Product Ownership vs. The Conveyor Belt Approach

Agile Metrics

To Estimate or Not to Estimate, Is that the Question? (2017 Better Software C...

Flow Efficiency - The most powerful improvement driver - Lean Kanban Central ...

Lead Time: What We Know About It...

Metrics at Every (Flight) Level [2020 Agile Kanban Istanbul FlowConf]

Agile Workshop: Agile Metrics

Metrics As A Learn And Change Agent

Clarity First: What it is. Why you need it. How to get it.

Improving Agility (Learning from Maersk Line's Journey) | Özlem Yüce | Agile ...

Value Stream Mapping: What to Do Before You Dive In

LKRU14 and LKAPAC: Five Numbers

A TP3 Approach to Manage Manufacturing Competitiveness After the Pandemic

The Importance of Estimating - WCMKE 2015

Top Metrics for Agile @Agile NCR2011

Value Stream Mapping: From Tool to Management Practice

Viewers also liked

Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...

Mike Villiger

Tech Mahindra and CollabNet have worked together on a number of mission-critical projects, and over the course of their partnership have developed unique expertise in lifecycle, development-to-production metrics. Gain an understanding not only of what metrics are important, but also practical approaches to building reports and dashboards that deliver a single-pane view of all your delivery pipelines across the enterprise. Participants will learn: KPI’s of end-to-end dashboard driven development and delivery Best practices for metrics in Agile / DevOps environments Role of technology frameworks for integrated planning and reporting

Metrics to Power DevOps

CollabNet

Boston DevOps Days 2016: Implementing Metrics Driven DevOps - Why and How

Andreas Grabner

DevOps Pipelines and Metrics Driven Feedback Loops

Andreas Grabner

Introducing DevOps

Nishanth K Hydru

Tech Talk: The Continuous Delivery Metrics That Will Make Your Pipeline Fast ...

CA Technologies

Welcome to the Future! Preparing for our agile testing journeys, Agile Testin...

lisacrispin

Trends in Agile Testing by Lisa Crispin

Directi Group

Lean DevOps Metrics

Bill Donaldson

Test automation_strategy_for_legacysystems

VodqaBLR

Introduction to Gauge

vodqancr

DevOps Gamification Workshop at JTEL Summer School 2015

IstvanKoren

Barely any modern software development project will be initiated without the discussion on test automation. With the demand to deliver software products in higher quality faster, test automation is oftentimes perceived by project leads as the silver bullet, allowing to ‘test everything’ without high overhead in employing skilled workforce. The tools are being discussed more often, than development and maintenance costs of automation frameworks that can support projects from inception into delivery and post-production. Incorrect approaches that result in choosing invalid test automation strategies are often the cause of much frustration later in the project, when the reality of test automation activities become visible and need to be accounted for. In this webinar, Anna will discuss her experiences in selecting short term and long term test automation strategies applied within various contexts, various skill sets that are required for running a successful test automation project, and suggest alternatives to full-blown test automation with the insufficient project resources. Learn: * when the test automation is an enabler, and when it’s an impediment to project success * when to start planning for automation, and how to select the tools and methods most suitable for specific project needs * how to anticipate investment costs and support long term automation effort across multiple projects * skills and roles in test automation

Winning strategies in Test Automation

XBOSoft

Agile 20015 Boot Camp: Intro to Agile Testing - Everyone Owns Quality

lisacrispin

Inner-Source: The Lesson of Linux for Enterprises

Samsung Open Source Group

Metrics-driven Continuous Delivery

Andrew Phillips

Deployment is the new build

Andrew Phillips

Inner Source 101 - GWO2016

Jim Jagielski

Open source is for life, not just for Christmas

Richard Lord

Companies can struggle to get their internal operations working smoothly. Cooperation between departments is essential. A silo mentality can easily occur when different groups or departments within an organisation do not share knowledge or information. This influences performance negatively and can contribute to a failing corporate culture. The proactive development of a user community and the use of open source collaborative strategies will greatly benefit any organisation. InnerSource takes the lessons learned from open source projects and applies them to the internal company community or culture. InnerSource can be a great tool to help break down silos, encourage collaboration and create a happier workplace.

[Srijan Wednesday Webinars] InnerSource: An Open Source Approach to Community...

Srijan Technologies

Viewers also liked (20)

Performance Metrics Driven CI/CD - Introduction to Continuous Innovation and ...

Metrics to Power DevOps

Boston DevOps Days 2016: Implementing Metrics Driven DevOps - Why and How

DevOps Pipelines and Metrics Driven Feedback Loops

Introducing DevOps

Tech Talk: The Continuous Delivery Metrics That Will Make Your Pipeline Fast ...

Welcome to the Future! Preparing for our agile testing journeys, Agile Testin...

Trends in Agile Testing by Lisa Crispin

Lean DevOps Metrics

Test automation_strategy_for_legacysystems

Introduction to Gauge

DevOps Gamification Workshop at JTEL Summer School 2015

Winning strategies in Test Automation

Agile 20015 Boot Camp: Intro to Agile Testing - Everyone Owns Quality

Inner-Source: The Lesson of Linux for Enterprises

Metrics-driven Continuous Delivery

Deployment is the new build

Inner Source 101 - GWO2016

Open source is for life, not just for Christmas

[Srijan Wednesday Webinars] InnerSource: An Open Source Approach to Community...

Similar to DevOps Metrics - Lies, Damned Lies and Statistics

Agile metrics can be used to the advantage or the detriment of teams and an organisation’s Agile success. This session looks at several of the core Agile metrics used to measure success to help you understand what success looks like, why the metric is desirable and what the metrics can tell us. Understanding why we want these metrics is critical to capturing something of value, rather than just doing 'because'. What will leaders and decision makers do with these metrics? What value do they add? Steve will also dive into the negative impacts of some of the Agile metrics we are sometimes forced to capture, how chasing velocity leads to gaming the system etc. He’ll look at bad metrics such as the seven deadly sins of Agile measurement and how to avoid them in your enterprise.

Steve Lawrence - Agile Metrics

AgileNZ Conference

How to Enter the Data Analytics Industry?

Ganes Kesari

Entering the Data Analytics industry

Gramener

Melbourne Business School - mba talk october 14 - croll - 40m - lean analytics

Lean Analytics

Optimizely Workshop: Take Action on Results with Statistics

Optimizely

By Srikant Kotapalli – VWO AB Testing has traditionally been one of the most commonly used methods to boost conversion rates. Many marketers have since used it tactically, that gets them some early wins but are unable to convert it into repeatable success. Building an effective optimization strategy needs an in-depth understanding of user behaviour, adopting a structured framework and constantly measuring and improving upon the results. This session will walk you through everything you need to know to transform your AB testing into a winning optimization engine.

Build a Winning Conversion Optimization Strategy

Savage Marketing

091 Process Mapping

Dr Fereidoun Dejahang

Prioritization and validation are key activities for successful product management. As a product manager, how will you succeed if you are not able to prioritize correctly? How do you determine what feature to release next and if it is the correct feature to build in the first place? This talk will cover principle methods and frameworks for feature validation and prioritization and is recommended for product managers and people working in product.

Validation and hypothesis based product management by Abdallah Al-Khalidi

Abdallah Al-Khalidi

Quality Management and NABH.pptx

ssuser10233b1

Five days of kaizen

Robert Simmons

What is the best way to measure progress on an Agile project?

Thoughtworks

LeanKaizenTraining

ahmad bassiouny

Awareness To Lean & 7 Qc Tools

Nilesh Sawant

Cause & Effect

Anand Subramaniam

New Employee Lean Orientation

flevko

howtoturnbigdataintobetterdecisionspauwelsemac2016

Koen Pauwels

Example of Marketing Research Analytics Project

Joao Rendon Kahn

Presentation by Lora Cecere at the Foundation of Strategic Sourcing

Lora Cecere

Data science toolkit for product managers

ProductFolks

Data Science Toolkit for Product Managers

Mahmoud Jalajel

Similar to DevOps Metrics - Lies, Damned Lies and Statistics (20)

Steve Lawrence - Agile Metrics

How to Enter the Data Analytics Industry?

Entering the Data Analytics industry

Melbourne Business School - mba talk october 14 - croll - 40m - lean analytics

Optimizely Workshop: Take Action on Results with Statistics

Build a Winning Conversion Optimization Strategy

091 Process Mapping

Validation and hypothesis based product management by Abdallah Al-Khalidi

Quality Management and NABH.pptx

Five days of kaizen

What is the best way to measure progress on an Agile project?

LeanKaizenTraining

Awareness To Lean & 7 Qc Tools

Cause & Effect

New Employee Lean Orientation

howtoturnbigdataintobetterdecisionspauwelsemac2016

Example of Marketing Research Analytics Project

Presentation by Lora Cecere at the Foundation of Strategic Sourcing

Data science toolkit for product managers

Data Science Toolkit for Product Managers

Recently uploaded

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & Dating Escorts Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

HyderabadDolls

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service Available 24/7 Hire Our agency presents a selection of young, charming call girls #J11 available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives,#J11 College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Kolkata ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 8005736733 🔝. for details Operating 24/7, we serve various locations in Kolkata, including Green Park, near metro stations. For premium call girl services in Kolkata 🔝 8005736733 🔝. Thank you for considering us!

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service...

HyderabadDolls

Yilin Xia (yilinx2@illinois.edu), Shawn Bowers (bowers@gonzaga.edu), Lan Li (lanl2@illinois.edu), and Bertram Ludäscher (ludaesch@illinois.edu) Presented at IDCC-2024 in Edinburg. ABSTRACT. We propose a new approach for modeling and reconciling conflicting data cleaning actions. Such conflicts arise naturally in collaborative data curation settings where multiple experts work independently and then aim to put their efforts together to improve and accelerate data cleaning. The key idea of our approach is to model conflicting updates as a formal argumentation framework (AF). Such argumentation frameworks can be automatically analyzed and solved by translating them to a logic program PAF whose declarative semantics yield a transparent solution with many desirable properties, e.g., uncontroversial updates are accepted, unjustified ones are rejected, and the remaining ambiguities are exposed and presented to users for further analysis. After motivating the problem, we introduce our approach and illustrate it with a detailed running example introducing both well-founded and stable semantics to help understand the AF solutions. We have begun to develop open source tools and Jupyter notebooks that demonstrate the practicality of our approach. In future work we plan to develop a toolkit for conflict resolution that can be used in conjunction with OpenRefine, a popular interactive data cleaning tool.

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...

Bertram Ludäscher

Aspirational Block Program Block Syaldey District - Almora

GovindSinghDasila

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort Service Available 24/7 Hire CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort S...

HyderabadDolls

Saudi Arabia [ Abortion pills) Jeddah/riaydh/dammam/+966572737505☎️] cytotec tablets uses abortion pills 💊💊 How effective is the abortion pill? 💊💊 +966572737505) "Abortion pills in Jeddah" how to get cytotec tablets in Riyadh " Abortion pills in dammam*💊💊 The abortion pill is very effective. If you’re taking mifepristone and misoprostol, it depends on how far along the pregnancy is, and how many doses of medicine you take:💊💊 +966572737505) how to buy cytotec pills At 8 weeks pregnant or less, it works about 94-98% of the time. +966572737505[ 💊💊💊 At 8-9 weeks pregnant, it works about 94-96% of the time. +966572737505) At 9-10 weeks pregnant, it works about 91-93% of the time. +966572737505)💊💊 If you take an extra dose of misoprostol, it works about 99% of the time. At 10-11 weeks pregnant, it works about 87% of the time. +966572737505) If you take an extra dose of misoprostol, it works about 98% of the time. In general, taking both mifepristone and+966572737505 misoprostol works a bit better than taking misoprostol only. +966572737505 Taking misoprostol alone works to end the+966572737505 pregnancy about 85-95% of the time — depending on how far along the+966572737505 pregnancy is and how you take the medicine. +966572737505 The abortion pill usually works, but if it doesn’t, you can take more medicine or have an in-clinic abortion. +966572737505 When can I take the abortion pill?+966572737505 In general, you can have a medication abortion up to 77 days (11 weeks)+966572737505 after the first day of your last period. If it’s been 78 days or more since the first day of your last+966572737505 period, you can have an in-clinic abortion to end your pregnancy.+966572737505 Why do people choose the abortion pill? Which kind of abortion you choose all depends on your personal+966572737505 preference and situation. With+966572737505 medication+966572737505 abortion, some people like that you don’t need to have a procedure in a doctor’s office. You can have your medication abortion on your own+966572737505 schedule, at home or in another comfortable place that you choose.+966572737505 You get to decide who you want to be with during your abortion, or you can go it alone. Because+966572737505 medication abortion is similar to a miscarriage, many people feel like it’s more “natural” and less invasive. And some+966572737505 people may not have an in-clinic abortion provider close by, so abortion pills are more available to+966572737505 them. +966572737505 Your doctor, nurse, or health center staff can help you decide which kind of abortion is best for you. +966572737505 More questions from patients: Saudi Arabia+966572737505 CYTOTEC Misoprostol Tablets. Misoprostol is a medication that can prevent stomach ulcers if you also take NSAID medications. It reduces the amount of acid in your stomach, which protects your stomach lining. The brand name of this medication is Cytotec®.+966573737505) Unwanted Kit is a combination of two medicin

Abortion pills in Jeddah | +966572737505 | Get Cytotec

Abortion pills in Riyadh +966572737505 get cytotec

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models...

nirzagarg

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

nirzagarg

Top profile Call Girls In Bihar Sharif [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Bihar Sharif [ 7014168258 ] Call Me For Genuine Mod...

nirzagarg

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Service Booking Now open +91- 9332606886 Pune VIP Call Girls is highly qualified professionals. #S10 They will offer only top quality services and never try to scam or drain you of money; in fact, they strive to give complete satisfaction so that each cent spent was worth its worth. Furthermore, these professionals pride themselves in keeping customer details confidential; any information gleaned will never be shared with any third party.$V15 Two shots with one girl: ₹4500/in-call, ₹7500/out-call Body to body massage with : ₹5500/in-call Full night for one person: ₹7000/in-call, ₹12000/out-call •𝐂𝐨𝐥𝐥𝐞𝐠𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐇𝐨𝐮𝐬𝐞𝐰𝐢𝐯𝐞𝐬 •𝐇𝐢𝐠𝐡 𝐏𝐫𝐨𝐟𝐢𝐥𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐑𝐚𝐦𝐩 𝐌𝐨𝐝𝐞𝐥𝐬 •𝐅𝐨𝐫𝐞𝐢𝐠𝐧𝐞𝐫 𝐆𝐢𝐫𝐥𝐬 𝐎𝐮𝐫 𝐞𝐬𝐜𝐨𝐫𝐭 𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 •𝐇𝐉 (𝐇𝐚𝐧𝐝 𝐉𝐨𝐛) •𝐁𝐉 (𝐁𝐥𝐨𝐰𝐣𝐨𝐛) # Completion # (Oral To Completion) # Special Massage # O-Level (Oral ) # Oral With A Noncondom) # COB (Come On Body) # Extraball (Have Many Times) # All Meetings 100%Safe

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

kumargunjan9515

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C 9548273370 Escort Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

HyderabadDolls

SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...

Elaine Werffeli

+97470301568 Qatar THC Oil and weed in Qatar? Where can I get THC vape in Doha Qatar?WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in France,Germany, Poland Serbia,Romania, Ukraine WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia Qatar Where can I get weed in Qatar? Where can I get THC vape in Doha Qatar?WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in France,Germany, Poland Serbia,Romania, UkraineWhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia Qatar WhatsApp+97470301568 WhatsApp +97470301568 Buy Weed, Cocaine, Heroin and Shrooms in Qatar Dubai UAE Malaysia Oman Kuwait Bahrain Saudi Arabia #Singapore #Jordan #Ireland, #Belgium, #United Kingdom, #Iceland, #*Portugal, Spain, China, Japan, Turkey, Canada United States, Morocco, France,Germany, Poland Serbia,Romania, Ukraine, and all countries United Arab Emirates . Our team has succesfully delivered in 26 different countries . All marijuana and Cocaine is double vacuum packed before shipping, making it completely odorless to ensure that it arrives safely to your door. Our distribution crew is expert at making packages that blend in with the rest of the mail. We have also put into place many other security measures to ensure the security of our customers. buy weed Dubai +97470301568buy Weed Qatar #Buy Weed Kuwait #Buy Weed Bahrain #Buy #Weed #Oman #Buy Weed UAE #Buy Weed Abu Dhabi @Buy Weed Doha Qatar #@Buy Weed Ajman #@Buy Weed Online #@Buy Weed UK #@Buy Weed Iceland #*@Buy Weed All Countries Below are the various strains of kush available ; buy hash and Weed in dubai,abu dhabi,sharjah where to buy weed in doha,where can i find weed in jeddah,Can I get weed delivered to Riyadh?,Buy weed Online Jeddah Saudi Arabia,Buy Weed and THC Cannabis Oil online ,QATAR , DOHA buy kush in DOHA , buy kush in DOHAWeed in QATAR # DOHA Buy Weed and THC Cannabis Oil online who delivers at your own location in Qatar Doha ,Kuwait ,Dubai including cannabis / weed,Where can I find weed in Dubai as a tourist?,Is marijuana allowed in Dubai ? How much is medical marijuana in Dubai ?Is weed legal in Dubai ?How to get marijuana in Saudi Arabia Do people in Saudi Arabia smoke weed ? Is Hash legal in Saudi Arabia ? Where is marijuana the most illegal?Can you get weed in Baku? Dubai, United Arab Emirates Canabis smokers - Dubai, Buy Marijuana Products Online in UAE Desertcart ships the Marijuana products in Dubai ,Abu Dhabi, Sharjah, Al Ain, Ajman and more cities in UAE. Get unlimited free shipping in 164+ countries Buy Weed Products Online in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Order thc Weed in Saudi Arabia Buy Marijuana Saudi Arabia weed in jeddahis cbd legal in saudi arabia cali tins smoke in saudi arabia drug use saudi arabia Buy weed marijuana. White Widow OG #Kush Sensi Star x ak 47 Afghan Ku

+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...

Health

20240412-SmartCityIndex-2024-Full-Report.pdf

khraisr

Top profile Call Girls In Latur [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Latur [ 7014168258 ] Call Me For Genuine Models We ...

gajnagarg

Abortion pills in Kuwait city !🌆+918761049707^) Where to get cytotec pills in salmiyah, WhatsApp '''(+918761049707)''' Explore Discreet and Safe Abortion Pill Options in Dubai, Empowering Women With Confidential and Medically Supervised Choices For Reproductive Health" Abortion Pills In Dubai, Abu Dhabi/Alain/Sharjah/RAK City Satwa-Mifepristone and Misoprostol Available in Dubai/Abu Dhabi - i Pills in Dubai and Cost Of Cytotec in WhatsApp (+918761049707) Abortion Pills in Dubai/Abu Dubai/ Abu Dhabi. Are you stranded with unwanted pregnancy in Dubai, Abu Dhabi , the United Arab Emirates (U.A.E), Qatar, Oman, Saudi Arabia or Kuwait? you can now contact us now on Whatsapp Dr AJ:+918761049707to buy safe abortion pills In Dubai, Abu Dhabi, Sharjah, Al Ain, Ajman, RAK City, Ras Al Khaimah and Fujairah to terminate an unwanted pregnancy in Dubai and the United Arab Emirates. Get your Discreet 100% Safe (+918761049707 )*Effective Abortion Pills For Sale in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE. BUY Mifepristone and Misoprostol (Cytotec), Mtp Kit In UAE. Abortion pills available in UAE (United Arab Emirates), Saudi Arabia, Kuwait, Oman, Bahrain and Qatar. Contact us today. +918761049707 -The UAE’s leading abortion care service in Dubai. Abortion Treatment. Medical Abortion. Surgical Abortion. Find A Clinic like Dr Maria Abortion clinic in Dubai We have Abortion Pills / Cytotec Tablets Available in Dubai, Abu Dhabi, KUWAIT, QATAR, BAHRAIN, DOHA, SALMIYA, Sharjah, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE., buy cytotec in Dubai abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain We sell original abortion medicine which includes: Cytotec 200mcg (Misoprostol), Mifepristone, Mifegest-kit, Misoclear, Emergency contraceptive pills, Morning after sex pills, ipills, pills to prevent pregnancy 72 hours after sex. All our pills are manufactured by reputable medical manufacturing companies like PFIZER. Medical abortion is easy and effective for everyone to perform in their own privacy. There are very few complications that may arise from medical abortion if one follows the right guidelines as instructed by the obstetrician. Abortion Pills in Dubai Can Now Be Offered at Dr Maria Abortion clinic in Dubai, F.D.A. Mifepristone and Misoprostol, the first of two drugs in medication abortions, previously had to be dispensed only by clinics, doctors or a few mail-order pharmacies like Dr Maria Abortion clinic in Dubai . Now, We can provide it. For the first time, retail pharmacies, like Dr Maria Abortion clinic in Dubai, will be Able to offer abortion pills in Dubai under a regulatory change made Tuesday by the Food and Drug Administration. The action could significantly expand access to abortion through medication. Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

ahmedjiabur940

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

gajnagarg

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We ...

nirzagarg

7. Epi of Chronic respiratory diseases.ppt

ibrahimabdi22

Ranking and Scoring Exercises for Research

Rajesh Mondal

Recently uploaded (20)

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

Jodhpur Park | Call Girls in Kolkata Phone No 8005736733 Elite Escort Service...

Reconciling Conflicting Data Curation Actions: Transparency Through Argument...

Aspirational Block Program Block Syaldey District - Almora

Nirala Nagar / Cheap Call Girls In Lucknow Phone No 9548273370 Elite Escort S...

Abortion pills in Jeddah | +966572737505 | Get Cytotec

Top profile Call Girls In Begusarai [ 7014168258 ] Call Me For Genuine Models...

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

Top profile Call Girls In Bihar Sharif [ 7014168258 ] Call Me For Genuine Mod...

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

Charbagh + Female Escorts Service in Lucknow | Starting ₹,5K To @25k with A/C...

SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...

+97470301568>>weed for sale in qatar ,weed for sale in dubai,weed for sale in...

20240412-SmartCityIndex-2024-Full-Report.pdf

Top profile Call Girls In Latur [ 7014168258 ] Call Me For Genuine Models We ...

In Riyadh ((+919101817206)) Cytotec kit @ Abortion Pills Saudi Arabia

Top profile Call Girls In bhavnagar [ 7014168258 ] Call Me For Genuine Models...

Top profile Call Girls In Satna [ 7014168258 ] Call Me For Genuine Models We ...

7. Epi of Chronic respiratory diseases.ppt

Ranking and Scoring Exercises for Research

DevOps Metrics - Lies, Damned Lies and Statistics

1. DevOps Metrics lies, damned lies and statistics Gaetano Mazzanti @mgaewsj agile42

2. why do we need metrics?

3. why do we need metrics? key reason: to improve

4. why do we need metrics? decisions

5. why do we need metrics? predictability

6. beware of cheating averages fallacies

7. beware metrics can be gamed

8. metrics & statistics still require reasoning and visual examination beware

9. same mean, variance & correlation (7.50 3.75 0.816)

10. use your eyes (same mean, variance & correlation) median = 7.58 median = 8.14 median = 7.11 median = 7.04

11. ecological fallacy average math score 50/100 60/100 group A 70% of people in A have better score than B! group B 70%

12. ecological fallacy 50/100 60/100 70% Group A Group B # score # score 50 45 70 43 50 55 30 100 group A group B average math score

13. exception fallacy

14. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live

15. Simpson’s paradox Global Natural Treat Live 108 153 Die 123 120 Natural 47% live Treat 56% live Women Natural Treat Live 57 32 Die 100 57 Natural 36,3% live Treat 36,0% live Men Natural Treat Live 51 121 Die 23 63 Natural 69% live Treat 45% live

16. which metrics?

17. deployment frequency lead time for changes mean time to recover change fail rate how IT performance was measured !?!?!?

18. ITIL KPIs “ITIL Key Performance Indicators (ITIL KPIs) are used to assess if the processes of an IT organization are running according to expectations” and if not… just kidding

19. a few ITIL KPIs… example (1/2)

20. a few ITIL KPIs… example (2/2)

21. it’s easy to get lost in a maze of (not relevant) data

22.

23. why? key question about your metrics

24. what do you want to learn? key question about your metrics

25. loops

26. improvement loops build/ change measurelearn experiment actionable metric hypothesis

27. which metrics matter to customers no yes end-to-end (global) functional (local) typical ideal

28. service oriented mindset DevOps as a service provided to deliver value to the business

29. pizza delivery fast delivery accuracy and quality predictability what matters to customers

30. let work ﬂow ﬂow is the movement and delivery of customer value through a process

31. derive from poor ﬂow slow delivery low quality unpredictability

32. poor ﬂow => queues

33. just 3 metrics? Work In Progress Lead Time Throughput

34. Little’s Law Items In Queue = Arrival Rate * Waiting Time Lead Time = Work In Progress / Throughput

35. focus on lead time

36. 0 1 2 3 4 5 6 7 8 1-Feb 3-Feb 5-Feb 7-Feb 9-Feb 11-Feb 13-Feb 15-Feb 17-Feb 19-Feb 21-Feb scatterplot 54% 71% 88% 96% lead time (days) average

37. scatterplot source ActionableMetrics book Lead

38. scatterplot (only bugs) source ActionableMetrics book Lead

39. frequency distribution source ActionableMetrics book Lead

40. Weibull distribution

41. what to aim for

42. aging source ActionableMetrics book

43. ___ efﬁciency process efﬁciency = total time active time________ 20 5 = 25% !1!!2!!3!!4!!5!!6!!7!!8!!9!10!11!12!13!14!15!16!17!18!19!20! elaborate do validate deliver waiting active

44. SLAs Service Level Agreements agreementexpectation a SLA is a contract between a service provider and the user/ customer that deﬁnes the level of service expected from the service provider i.e. we expect an item to ﬂow through the process and exit in 5 days or less with an 85% probability of success

45. SLAs – some hints do not set a SLA without analyzing Lead Time data do not allow a SLA to be set by someone external to your group do not set a SLA without collaborating with customers and/or other stakeholders use different SLAs for different Work Item Types

46. SLA

47. slack – avoid full utilization absorb variations % capacity utilization queuesize queue size grows exponentially at high capacity0 5 10 15 20 25 0 10 20 30 40 50 60 70 80 90 100

48. your policies shape your data your data shape your policies

49.

50.

51.

52.

53.

54.

55. where to start from?

56. a possible approach understand sources of dissatisfaction analyze demand & capacity discover work item types measure ﬂow set SLAs setup metric based improvement experiments (similar to STATIK…)

57. obsessions you should have improve process continuously remove problems/impediments asap get help from metrics

58.

59. Gaetano Mazzanti @mgaewsj gaetano.mazzanti@agile42.com agile42

DevOps Metrics - Lies, Damned Lies and Statistics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to DevOps Metrics - Lies, Damned Lies and Statistics

Similar to DevOps Metrics - Lies, Damned Lies and Statistics (20)

More from Gaetano Mazzanti

More from Gaetano Mazzanti (19)

Recently uploaded

Recently uploaded (20)

DevOps Metrics - Lies, Damned Lies and Statistics