Measuremen Systems Analysis Training Module

1 January 13, 2017 – v2.0
Six Sigma Measurement Systems Analysis
by Operational Excellence Consulting LLC

2 January 13, 2017 – v2.0
Section 1: Introduction
Section 2: Sources of Variation
Section 3: Measurement System Terminology
 Relevance
 Discrimination
 Accuracy
 Stability
 Linearity
 Precision
Section 4: Variable Gage R&R Study
Section 5: Attribute Gage R&R Study
Measurement Systems Analysis – Table of Contents

3 January 13, 2017 – v2.0
The Science of Measurement
“I often say that when you measure what you are
speaking about and express it in numbers, you
know something about it.”
Lord Kelvin, 1891

4 January 13, 2017 – v2.0
Why Measurement Systems Analysis?
• An evaluation of the measurement system MUST be undertaken to
ensure effective analysis of any subsequent data generated for a
given process or product characteristic.
Observed Value = Master or True Value + Measurement Error
• Measurement error is a statistical term meaning the net effect of all
sources of measurement variability that cause an observed value to
deviate from the master or true value.
• The key question that a measurement systems analysis seeks to
answer is:
Can you trust your data?

5 January 13, 2017 – v2.0
What is a Defect ?
A defect is any variation of a required product or
process characteristic, which is far enough removed
from its nominal value to prevent the product or
process from fulfilling the physical and functional
requirements of the internal or external customer.
Measurement Systems Analysis – A Defect Definition

6 January 13, 2017 – v2.0
Do we know if the excessive variation comes from the
process/product or from the measurement system ?
Upper
Specification
Limit (USL)
Lower
Specification
Limit (LSL)
Measurement Systems Analysis – A Defect Definition
Defects
Defects

7 January 13, 2017 – v2.0
 Relevance
 Discrimination
 Accuracy
 Stability
 Linearity
 Precision

8 January 13, 2017 – v2.0
Possible Sources of Measurement Variation
Actual Process
Variation
Long Term
Process Variation
Short Term
Process Variation
Measurement System
Variation
Variation due
to Operators
Variation
within Operator
RepeatabilityReproducibility
Observed Measurement
Variation
To address actual process variability, the variation due to
the measurement system must first be identified,
quantified and possibly reduced.

9 January 13, 2017 – v2.0
Measurement Systems Analysis – Objectives
• Obtain information relative to the amount and type of measurement
variation associated with the measurement system
• Criteria to accept and release new measuring equipment
• Comparison of measuring device against another
• Basis for evaluating a gage suspect of being deficient
• Measurement Systems Terminology
– Relevance
– Discrimination
– Accuracy
– Stability
– Linearity
– Precision

10 January 13, 2017 – v2.0
 Relevance
 Discrimination
 Accuracy
 Stability
 Linearity
 Precision

11 January 13, 2017 – v2.0
Measurement System Relevance
• A measurement system is relevant if it measures a product or process
quality characteristic that is truly important to the customer
• The measurement system measures the “practical problem” to be
solved
• A relevant measurement system is able to distinguish between what
the customer sees as definitely “good” or “bad” product
• A relevant measurement system does not have to mimic customer use
exactly, as long as the “failure mode” remains the same
• Relevance is the most critical property of a
measurement system or test method

12 January 13, 2017 – v2.0
Doing the right things first !
Focusing and solving
the “right” problem !!!
OK FAIL
OK
FAIL
PRODUCT/
PROCESS
MEASUREMENT
SYSTEM
We often think only in the categories A (ok) or D (fail) and neglect the also
important categories B and C. Category B means that the process (e.g.
measurement process) is insufficient with respect to the specifications.
Category C means that the product itself (e.g. specifications, material) is
insufficient with respect to the customer specifications.
A B
C D

13 January 13, 2017 – v2.0
Measurement System Discrimination
• The number of decimal places that can be measured by the system
• Increments of measure should be about 1/10 of the width of the
specification or process variation
• Poor discrimination may be thought of as excessive round-off error
3 . 1 6 2
140
130
120
.2 .6
Digital Readout Resolution: 0.001 Unit
Dial Indicator may be 0.1 Unit or less if rounded up or down
Strip Chart may be 10 units or less if rounded up
or down

14 January 13, 2017 – v2.0
Measurement System Discrimination - Example
• The dot plot of the Gage A results above show 5 different results across the
range of process variation from 1 to 5. Therefore, the discrimination of Gage A
is 5.
• The dot plot of the Gage B results above show 9 different results across the
range of process variation from 1 to 5. Therefore, the discrimination of Gage B
is 9.
• A discrimination of 10 or more is desired, but of course not always achievable.
54321
Gage A
Dotplot of Gage A
4.84.23.63.02.41.81.2
Gage B
Dotplot of Gage B

15 January 13, 2017 – v2.0
Measurement System Accuracy
• Measurement System Accuracy is the difference between the
observed average of the measurements and the true average.
• Establishing the true average is best determined by measuring with
the most accurate measuring equipment available.
• True Value
– Theoretically correct value
– NIST standards
• Bias
– Distance between observed average value of all measurements and the
true value
– Amount the measurement device is consistently off target
– Systematic error or offset

16 January 13, 2017 – v2.0
Measurement System Accuracy - Example
• Let’s assume the true value of what is measured is 10. Then, Gage 2 provides in
average the correct result and is therefore accurate. However, Gage 1 measures in
average about 5 too low and Gage 3 measures in average 5 too high. Therefore, Gage
1 has a bias of “-5” and Gage 3 has a bias of “+5”. Both Gages 1 and 3 are inaccurate.
1
0.0
1.0
2.0
3.0
4.0
5.0
2 4 6 8 01 21 41 6
5
NveDtSnaeM
0013269.040.51
001460.1489.9
001351.1240.
D
ytisneD
ata
G
elbairaV
3egaG
2egaG
1ega
H 3egaG,2egaG,1egaGfomargotsi
True Value
BiasBias

17 January 13, 2017 – v2.0
Measurement System Stability
• Measurement System Stability refers to the difference in the average of at
least two sets of measurements obtained with the same gage on the same
parts taken at different times.
• Defined as the distribution of measurements that remains constant and
predictable over time for both the mean value and the standard deviation.
• Evaluated using a control chart for the MSA metrics.
• “A ten today is a ten tomorrow.”
Poor StabilityGood Stability

18 January 13, 2017 – v2.0
Measurement System Stability - Example
• A refractometer instantly reads gravity, in Brix and gravity, of unfermented
wort or fruit juice by measuring the degree that light passing through the
sample is bent. Unlike a hydrometer, only a few drops are required for a
sample. To use, apply 2-3 drops to the prism face, close cover, and look
through the eyepiece while aiming your refractometer at a light source.
• This refractometer for example measures from
0-32 Brix with an accuracy of +/- 0.2 Brix. It
includes automatic temperature compensation
for temperatures between 50-86 F, which
eliminates the need to consult temperature
correction charts.
• Why is a temperature correction charts
needed? - A refractometer is not stable over
time if the ambient temperature changes,
unless the gage corrects itself.

19 January 13, 2017 – v2.0
Measurement System Linearity
• Measurement System Linearity is the difference in the accuracy values
through the expected operating range.
• A gage linearity study assesses how accurate your measurements are
through the expected range of normal process variation.
• The inconsistency of the bias across the sizes indicates that the
measurement system has linearity problems.
True Value
ObservedValue
No Bias
Bias

20 January 13, 2017 – v2.0
Measurement System Linearity - Example
14121086420
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0
Y = Metrics
Bias
Scatterplot of Bias vs Y = Metrics
• The graph above indicates that the bias or inaccuracy of the measurement
system or gage varies over the range of measurements. Therefore, the
gage is not linear over the range of normal process variation.
Bias = 0
Bias > 0
Bias < 0

21 January 13, 2017 – v2.0
Measurement System Precision: Repeatability
• The inherent variability of the measurement system
• Repeatability is the variation that occurs when repeated measurements
are made of the same variable under as close to identical conditions as
possible
– Same operator
– Same measurement device
– Same samples
– Same environmental conditions
– …
• Repeatability is estimated by the pooled standard deviation of the
distribution of repeated measurements
• Repeatability is always less than the total variation of the system
rpt

22 January 13, 2017 – v2.0
Measurement Systems Repeatability
• The variation between successive measurements of the same
sample, same characteristic, by the same person using the same
instrument
• Also known as test - retest error; used as an estimate of short-term
variation
Worse Repeatability Better Repeatability

23 January 13, 2017 – v2.0
Measurement System Repeatability - Example
• Let’s assume that Operator A and B are measuring the same process outcome
with the same gage again and again. Then, the graphs above indicate that
Operator A has more variation in his/her measurements than Operator B. That
means that Operator B’s repeatability is better than Operator A’s.
3
0.0
1.0
2.0
3.0
4.0
51 81 12 42 72 0
2
NveDtSnaeM
052350.120.02
052350.313.0
D
ytisneD
ata
O
elbairaV
BrotarepO
Arotarep
H BrotarepO,ArotarepOfomargotsi

24 January 13, 2017 – v2.0
Measurement System Precision: Reproducibility
• Reproducibility is the variation that results when different conditions
are used to take the measurements. For example
– Different operators
– Different measurement devices
– Different environmental conditions
– …
• Different operator is the most common condition tested
• Reproducibility is estimated by the standard deviation of the averages
of measurements taken under the different measurement system
conditions
• Reproducibility is always less than the total variation of the system
rpd

25 January 13, 2017 – v2.0
Measurement Systems Reproducibility
• The difference in the average of the measurements made by different
persons using the same or different instrument when measuring the
identical product or process characteristic.
Worse Reproducibility Better Reproducibility

26 January 13, 2017 – v2.0
Measurement System Reproducibility - Example
• Let’s assume that Operator 1 and 2 are measuring the same product or process
outcome with the same gage again and again. Then, the graph above indicates
that while Operator A and Operator B have the same repeatability (i.e. variation),
the mean value of Operator 1’s measurements is significant lower than Operator
2’s. That means that the measurement system is not reproducible.
2
0.0
1.0
2.0
3.0
4.0
0.21 5.31 0.51 5.61 0.81 5.91 0.1
1
NveDtSnaeM
052400.188.91
052750.129.4
D
ytisneD
ata
O
elbairaV
2rotarepO
1rotarep
H 2rotarepO,1rotarepOfomargotsi

27 January 13, 2017 – v2.0
Precise, not accurate Accurate, not precise
Not Precise, not accurate Precise, accurate
Measurement System Accuracy vs. Precision

28 January 13, 2017 – v2.0
 Relevance
 Discrimination
 Accuracy
 Stability
 Linearity
 Precision

29 January 13, 2017 – v2.0
Variable Gage Repeatability & Reproducibility Study
Actual Process
Variation
Long Term
Process Variation
Short Term
Process Variation
Measurement
Variation
Variation due
to Operators
Variation
within Operator
RepeatabilityReproducibility
Observed Measurement
Variation
? ?
??
The different sources of variation in a process or
product characteristic.

30 January 13, 2017 – v2.0
Standard Deviations Do Not “Add Up” !
Process Step 1 Process Step 2
Let’s assume we have a process that consists of two
sequential processing steps.
Process Step 1 has an average lead time of 15 minutes
and a standard deviation of 3 minutes.
Process Step 2 takes a little bit longer and has an average
lead time of 25 minutes. It also has more variation with a
standard deviation of 4 minutes.
Let’s also assume that both processing steps create
normally distributed data with respect to lead time.
The table on the right shows some of the actual data for this example.
What is the average lead time and standard deviation of the end-to-end process?

31 January 13, 2017 – v2.0
The graphs above show the data when performing both processing steps
10,000 each.
We can see, that the average lead times and the standard deviations for
both samples are “pretty” close to the actual average lead time and
standard deviation of the processing steps as defined on the previous slide.

32 January 13, 2017 – v2.0
The graphs above show the sum of the lead times from both processing
steps.
We can see, that the total average lead times of the combined process
seems to be 40 minutes (15 + 25 Minutes. Makes sense!). However, the
standard deviation of the combined process seems to be 5 minutes.

33 January 13, 2017 – v2.0
Standard deviations do not add up. However, if we square the standard
deviations, then it makes all sense.
Remember the Pythagorean Theorem? Pretty cool. Right?
+≠
52
= 32
+ 42
𝑠1+2 = 𝑠1 + 𝑠2
2 2 2
25 = 9 + 16
→

34 January 13, 2017 – v2.0
Let’s apply that to what is called a Gage Repeatability & Reproducibility
Study or short a Gage R & R Study.
The standard deviation we see in our measurement results, i.e. observed
process variation, is caused by the standard deviation of the different
process outcomes or parts, i.e. true process variation, and the standard
deviation introduced by the measurement system, i.e. lack of repeatability
and reproducibility. That means …
2
observed process variation
= 2
true process variation + 2
measurement system
Concepts of a Variable Gage R&R Study

35 January 13, 2017 – v2.0
The standard deviation introduced by the measurement system is due to
the lack of repeatability and reproducibility. That means …
2
measurement system
= 2
repeatability + 2
reproduciability
Or
2
observed process variation
= 2
true process variation + 2
repeatability + 2
reproduciability

36 January 13, 2017 – v2.0
To determine these different standard deviations, we need to perform a
special experiment, called a Gage Repeatability & Reproducibility
Study, or short Gage R&R Study.
• observed process variation
• true process variation
• measurement system
•repeatability
•reproducibility

37 January 13, 2017 – v2.0
A Gage R&R Study Metrics
One of the key results from a Gage R&R Study is the % R&R Ratio. The % R&R
Ratio is used to express the fitness of a measurement system to analyze and
determine sources of variation in the performance of the process characteristic.
measurement system
observed process variation
* 100%%R&R Ratio =
Fitness Test:
• Excellent measurement systems have a % R&R Ratio below 10%
• Adequate measurement systems have a % R&R Ratio below 30%
(but can be improved)
• Measurement systems with a % R&R Ratio greater than 30% need
to be improved before one can move forward with an improvement
project.

38 January 13, 2017 – v2.0
Variable Gage R&R Study Design
 Process Variation
• Include typically 10 – 15 samples to introduce process variation.
• Make sure that the samples cover at least 80% of the full, typical range of
the process variation. Otherwise, your %Gage R&R Ratio will be higher
than it actually is.
 Reproducibility Variation
• Include typically 2 – 3 operators to introduce reproducibility variation.
 Repeatability Variation
• Have each operator measure each part or process outcome at least twice
(replications) to introduce repeatability variation.
 Gage R&R Study Design Summary
# of Measurements = # of Samples x # of Operators x # of Replications

39 January 13, 2017 – v2.0
Variable Gage R&R Study Worksheet Design
• Using Minitab 17, we will first design a Gage R&R Study Worksheet to
create the desired design and record the results.
Enter the number of
products or process
outcomes
10
3
2
Enter the number
of operators
Enter the number
of replications
You can enter the
assessors or
operators actual
names and also
sample ‘name’
information

40 January 13, 2017 – v2.0
Performing a Variable Gage R&R Study
On the left you see the resulting work-
sheet that Minitab created based on the
information provided.
Column C4 has been added separately to
record the results of the measurements.
The key steps of performing a Variable
Gage R & R Study are described on the
next slide.

41 January 13, 2017 – v2.0
Performing a Variable Gage R&R Study
1. Calibrate the gage(s), or assure that it has or they have been calibrated.
2. Operator 1 measures all samples once in random order
3. Operator 2 measures all samples once in random order
4. Continue until all operators have measured each sample once
5. Repeat steps 2 - 4 for the required number of replications
6. Perform a graphical analysis of the Gage R&R Study results
1. Lack of Repeatability Issues
2. Lack of Reproducibility Issues
7. Perform a numerical analysis of the Gage R&R Study results
1. Standard deviations sobserved process variation , strue process variation , smeasurement system ,
srepeatability , and sreproducibility
2. %R&R Ratio
8. Determine next steps and follow-up action (if any)

42 January 13, 2017 – v2.0
Graphical Analysis of a Variable Gage R&R Study
Enter the columns that have the
 Part Number
 Operators
 Results / Measurement Data
Information of your Gage R&R Study
The first graphical tool we will use to analyze a Variable Gage R & R Study results
with Minitab is the Gage Run Chart.

43 January 13, 2017 – v2.0
For a good measurement system, only a small amount of variation will be seen
“within” each sample, i.e. Lack of Repeatability and Reproducibility.
Lack of
Repeatability
Lack of
Reproducibility

44 January 13, 2017 – v2.0
The second graphical tool we will use to analyze a Variable Gage R & R Study
results with Minitab is the Gage R&R Study (Crossed) Chart. Actually this will
create 6 graphs for us to analyze.
 Part Number
 Operators
 Results / Measurement Data

45 January 13, 2017 – v2.0
Lack of
Repeatability
Lack of
Reproducibility
Part-to-Part or
Process Variation

46 January 13, 2017 – v2.0
• A line is plotted for each operator by sample (average of all replications)
• Parallel lines, or better identical lines, are desired for all operators
• Interactions between operators are indicated by crossing lines, or lines that do
not remain parallel
• We need to understand and resolve “Operator – Sample” interactions where
they exist
Graphical Analysis – Lack of Reproducibility
Lack of
Reproducibility

47 January 13, 2017 – v2.0
Graphical Analysis – Lack of Repeatability
Lack of
Repeatability
• The Range chart should be ‘in control’
• There should be at least 5 possible
values within the control limits to
indicate adequate discrimination
• See the Discrimination Index reported
in the Session window
• In this X-bar chart we want to see variability
outside the control limits (red lines), to indicate
variability between samples
• If not, the study may not include appropriate
samples that cover the full range of typical
output variation

48 January 13, 2017 – v2.0
Graphical Analysis – Process vs. MS Variation
• The average (plus sign in the circle) and spread of the values for each sample is shown
by this graph
• Minimal spread for each sample, but variability between samples (means shifting) is here
desired, as it indicates large Part-to-Part variation, compared to the variation caused by
the measurement system
Part-to-Part or
Process Variation
Measurement
System Variation

49 January 13, 2017 – v2.0
Numerical Analysis – % Gage R&R Ratio
Total % Gage R&R Metrics a.k.a. %R&R Ratio
• Excellent measurement systems have a % R&R Ratio below 10%
• Adequate measurement systems have a % R&R Ratio below 30
• Measurement systems with a % R&R Ratio greater than 30% need to be
improved before one can move forward with an improvement project.
%R&R

50 January 13, 2017 – v2.0
Numerical Analysis – Distinct Categories
• We have six “buckets”, or six Distinct Categories that we can use
to classify measurements
• BE CAREFUL - watch for low Distinct Categories! Must be at
least 5 for Process Improvement use!

51 January 13, 2017 – v2.0
Ways to Improve Measurement Systems
 Sense Multipliers
 Devices to improve human senses
 Masks / Templates
 Block out unimportant information
 Checklists
 Organization or Reorganization of Work Area
 Visual Aids – Limit Samples – Golden Samples
 Automation and Sensors
 Product Redesign
 …

52 January 13, 2017 – v2.0
 Relevance
 Discrimination
 Accuracy
 Stability
 Linearity
 Precision

53 January 13, 2017 – v2.0
Concept of an Attribute Gage R&R Study
 Use good experimental practices
 Designate one person to conduct the study
 Remaining team members will be inspectors
 There must be at least 3 good or 3 bad samples, all can’t be good or
bad
 The ideal Attribute Gage R&R Study would contain 30-50 samples (!),
with at least 10 good and 10 bad samples

54 January 13, 2017 – v2.0
Performing an Attribute Gage R&R Study
1. Choose the samples randomly. Make sure the sample includes at
least 3 good or 3 bad samples.
2. First assessor (or operators) should inspect each sample in random
order and determine whether it is “acceptable” or “unacceptable”.
3. Record results for each sample. Keep samples separate - you will
need to know which results came from which sample.
4. The second, then the third assessor inspects each sample in a
random order.
5. After all assessors have assessed each samples once, Step 2 to 4
will be repeated at least once.

55 January 13, 2017 – v2.0
Attribute Gage R&R Study – An Example
In this example:
– There are 10 samples (more would be better)
– 3 different assessors or operators
– Each assessor evaluated each sample at twice
1 Good Good Good Good Good Good
2 Bad Bad Good Good Bad Bad
3 Good Bad Bad Bad Good Good
5 Bad Bad Bad Bad Bad Bad
7 Good Good Good Good Bad Good
9 Good Bad Good Good Good Good
10 Bad Bad Bad Bad Bad Bad
Assessor 1 Assessor 2 Assessor 3
Minitab
Worksheet
(using above data)

56 January 13, 2017 – v2.0
Analysis of an Attribute Gage R&R Study
 Measurement or Rating Results
 Part Numbers
 Assessor or Operator

57 January 13, 2017 – v2.0
Within Appraisers
Assessment Agreement
Appraiser # Inspected # Matched Percent 95 % CI
1 10 8 80.00 (44.39, 97.48)
2 10 10 100.00 (74.11, 100.00)
3 10 9 90.00 (55.50, 99.75)
# Matched: Appraiser agrees with him/herself across trials.
Fleiss' Kappa Statistics
Appraiser Response Kappa SE Kappa Z P(vs. > 0)
1 Bad 0.58333 0.316228 1.84466 0.0325
Good 0.58333 0.316228 1.84466 0.0325
2 Bad 1.00000 0.316228 3.16228 0.0008
Good 1.00000 0.316228 3.16228 0.0008
3 Bad 0.78022 0.316228 2.46727 0.0068
Good 0.78022 0.316228 2.46727 0.0068
Between Appraisers
Assessment Agreement
# Inspected # Matched Percent 95 % CI
10 6 60.00 (26.24, 87.84)
# Matched: All appraisers' assessments agree with each other.
Percentage of samples an appraiser
agreed with himself/herself.
Would like all % Agreements to be
at least 90%.
More about the CI (Confidence
Intervals on the next slide).
Percentage of samples all
appraisers agreed with each other.
Would like all % Agreements to be
at least 90%.
More about the CI (Confidence
Intervals on the next slide).

58 January 13, 2017 – v2.0
Percentage of samples an
appraiser (here Appraiser 1)
agreed with himself/herself.
Confidence Interval, indicating
the actual % Agreement for
Appraiser 3 with a 95% certainty
or confidence.

59 January 13, 2017 – v2.0
Measurement Systems Analysis – Summary
 An evaluation of the measurement system MUST be undertaken to ensure
effective analysis of any subsequent data generated for a given process or
product characteristic, i.e. the Y.
 Make sure that the measurement system provides accurate results, means it
has been properly calibrated
 Investigate any stability or linearity issues of your measurement system
 Make sure that the measurement system has enough discrimination to be
useful in determining different levels in the attribute being measured
 A measurement error is an inherently part of the total variation routinely
observed in the process or product characteristic Y
 A Gage R&R Study will determine the percentage of total variation caused by
the measurement system
 The goal is to minimize the controllable error in the measurement system
 …

60 January 13, 2017 – v2.0
The End …
“Perfection is not attainable, but if we chase perfection we can catch
excellence.” - Vince Lombardi

61 January 13, 2017 – v2.0
Terms & Conditions
After you have downloaded the training material to your own
computer, you can change any part of the course material and
remove all logos and references to Operational Excellence
Consulting. You can share the material with your colleagues and
re-use it as you need. The main restriction is that you cannot
distribute, sell, rent or license the material as though it is your
own. These training course materials are for your — and
your organization's — usage only. Thank you.

Measuremen Systems Analysis Training Module

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Measuremen Systems Analysis Training Module

Similar to Measuremen Systems Analysis Training Module (20)

More from Frank-G. Adler

More from Frank-G. Adler (14)

Recently uploaded

Recently uploaded (20)

Measuremen Systems Analysis Training Module