Scale development

Scale Development- An Overview
Michael Sony
emailofsony@gmail.com
Workshop on Research Methods at GUDMS

Measurement
Fundamental Activity in Science
• Physical Measures
• Examples
– Weight
– Pressure
– Voltage
– Temperature
– Speed

Measurement Scales

Measurement
Social Measures
• Examples
– Job Satisfaction
– Stigma perceived by people with HIV
– Happiness
– Quality of Life
– Motivation
– Emotional Intelligence
– Meaning in life

Meaning in life scale
_____1. I understand my life’s meaning.
_____2. I am looking for something that makes my life feel
meaningful.
_____3. I am always looking to find my life’s purpose.
_____4. My life has a clear sense of purpose.
_____5. I have a good sense of what makes my life meaningful.
_____6. I have discovered a satisfying life purpose.
_____7. I am always searching for something that makes my life
feel significant.
_____8. I am seeking a purpose or mission for my life.
_____9. My life has no clear purpose.
_____10. I am searching for meaning in my life.
Steger, M. F., Frazier, P., Oishi, S., & Kaler, M. (2006). The meaning in life questionnaire: Assessing
the presence of and search for meaning in life. Journal of counseling psychology, 53(1), 80.

Steps in Scale Development
Construct Definition
 Item Generation
Content Validity & Pretesting
Measurement Purification
Verification of Dimensionality
Nomological Validity
 Criterion related Validity
Accounting for Known issues in measurement
scales

Construct Definition

• Construct Definition
• Specify Domain of the Construct
• What is included and what is excluded
• Theory as an aid to clarity

Item Generation

Number of Items
• Impossible to answer
• Thumb rules Devellis (2003) – 3 to 4 times
large than final scale.
• Thumb rule for final scale for each dimension
minimum 3 items.

An Example
SERVQUAL
• Initial Dimension – 10
• Initial Item Generation -97
• Final Dimensions – 5
• Number of items – 22
– Tangibles – 4
– Reliability -5
– Responsiveness -4
– Assurance – 4
– Empathy -5
– % Reduction in items = 77%
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). Servqual. Journal of retailing,
64(1), 12-40. (Citation: 16407)

Characteristics of good or bad item
a) Exceptionally lengthy items
• It is fair to say that one of the things I seem
to have a problem with much of the time is
getting my point across to other people.
• I often have difficulty making a point.
b) Reading difficulty

c) Redundancy
– I will do almost anything to ensure my Childs
success
– No sacrifice is to great too great if it helps my
child achieve success

c) Redundancy
- In my opinion pet lover are kind people
- In my estimation pet lovers are kind people
• Redundancy is tolerated in initial pool of items

d) Multiple Negatives
• I am not in favor of corporations stopping
funding for antinuclear groups
• I favor continued private support of groups
advocating a nuclear ban

e) Double barreled
• I support civil rights because discrimination
is a crime against God.
• If a person supports civil rights for reasons
other than its affront to a deity (e.g.,
because it is a crime against humanity), how
should he or she answer?

f) Positively and Negatively Worded Items
- intent of wording items both positively and
negatively within the same scale is usually to
avoid an acquiescence, affirmation, or
agreement

Content Validity

• the degree to which an instrument has an
appropriate sample of items for the construct
being measured

Content Validity
1. Initial screening of items;
2. Expert assessment of the applicability of the
items to each dimension; and
3. Expert assessment of representativeness of
the items to each dimension

Applicability of the items to each
dimension
• Judges are given the definition of each dimension, a
related explanation and an example item
• The judges are asked to allocate the items to one of
the dimensions or to a “not applicable” category.
• Items were eliminated that did not receive the
appropriate category from at least (n-1) of the n
judges.
• Calculate correlations among Judges and inter judge
reliability

Technique to Spot Aberrant
Judges
• Intentional Introduction of lie items
• Random
• Camouflage
• Lie Item Detection rate: 80-100% Excellent
60 -80% Good
40 -60 % Average
< 40 % Poor

Representativeness of the items to
each dimension
• Each item is judged for Relevance , Simplicity
Clarity
• Calculate content validity index

Rating Form

CVI
Source: Polit and Beck,2006
Polit, D. F., & Beck, C. T. (2006). The content validity index: are you sure you know what's being reported? Critique
and recommendations. Research in nursing & health, 29(5), 489-497. Workshop on Research Methods at GUDMS

Guidelines
• Lynn’s (1986) criteria (I-CVI = 1.00 with 3 to 5
experts and a minimum I-CVI of .78 for 6 to 10
experts) and it would have an SCVI/ Ave of .90
or higher.
Lynn, M.R. (1986). Determination and quantification of content validity. Nursing Research,
35, 382– 385

Guidelines
• two rounds of expert review
• if the initial I-CVIs suggest the need for
substantial item improvements, or if the
reviewers identify aspects of the construct not
adequately covered by the initial pool of items
(Polite et al. 2006).
Polite, D. F., Beck, C. T., & Hungler, B. D. (2006). Essentials of Nursing Research;
methods, appraisal and uitilization.

Face Validity
• Face validity is a post hoc assessment of
whether the items in a scale measure a
construct.
• Experts from the field may be asked to
comment on the scale and appropriate
suggestion may be included

Pre-testing /Pilot Testing
• Representative target groups
• Pre-testing is the administration of the data
collection instrument with a small set of
respondents from the population.
• The purpose of pre-testing is to identify
problems with the data collection instrument
and find possible solutions.
• Recommended sample Size 10% of final
sample size.

Measurement Purification

Measurement Purification
• Exploratory factor analysis
• Conducted to provide some insight into the
dimensionality of the scale & Item reduction.

Measurement Purification- A
Guidelines
• Computation of coefficient of Cronbach Alpha
and Item to total correlation
• Deletion of those items whose items to total
correlation is low and its deletion increased
coefficient of Alpha
• Factor Analysis for data reduction and to
unearth underlying factor structure

Verification of Dimensionality

Verification of dimensionality
Construct Validity : the degree to which a
scale measures what it claims, or purports, to
be measuring
1. Convergent Validity
2. Discriminant Validity

• Convergent Validity:- The items that are
indicators of a same construct should
converge or share high proportion of variance.
• Average Factor loadings > 0.7

• Average factor loading for attitude towards co
workers = 0.87 , VE = 0.75
• Average factor loading for Environmental
Perception =0.835 VE = 0.70
• Average factor loading for Job Satisfaction
=0.825 , VE = 0.68

• Discriminant validity:- is the extent to which a
construct is truly distinct from other construct.
Guideline
• Variance extracted percentages for any two
construct > square of the correlations
estimate between these two constructs.

Discriminant Validity
• AC and EP = (0.75 + 0.70 )/2 = 0.72 = 0.72
• Correlation among them = 0.2252 = 0.05

Assessment of Reliability

Internal Consistency
• Internal consistency reliability analysis is a
parametric procedure used to evaluate the
consistency of results across items within a
single scale (i.e., instrument) or subscale that is
composed of multiple items.
• Cronbach α > 0.7

Assessment of reliability
Test – Retest
• Correlation of the scale with same
respondents at two different time points e.g.
after 4 weeks > 0.7

Nomological validity

Theoretical Level: Concepts, Ideas
Construct
Theoretical Level: Concepts, Ideas
Construct
Construct
Construct Construct

Nomological validity
• Examining Whether the correlations among
related consequences constructs make sense.
An Example
• Relationship Market Orientation and Business
Performance.
• Well grounded Theoretical reasons

Sin, L. Y., Tse, A. C., Yau, O. H., Chow, R. P., Lee, J. S., & Lau, L. B. (2005). Relationship
marketing orientation: scale development and cross-cultural validation. Journal of
Business Research, 58(2), 185-194.

Criterion related validity
Concurrent Validity Criterion Validity

• Concurrent validity :This measures the relationship
between measures made with existing tests. The
existing tests is thus the criterion.
• Concurrent validity is demonstrated when a test
correlates well with a measure that has previously
been validated ( Bivariate Correlation)
• E.g. Scores on a written first aid exam are highly
correlated with scores assigned by raters during a
hands-on measure in which examinees demonstrate*
*Salkind, N. J. (Ed.). (2010). Encyclopedia of research design. Sage.

• Predictive Validity: This measures the extent
to which a future level of a variable can be
predicted from a current measurement.
• This includes correlation with measurements
made with different instruments ( Regression
analysis)

Known Issues in Scale Development

Social Desirability Bias
• Social desirability bias is a social science
research term that describes the tendency of
survey respondents to answer questions in a
manner that will be viewed favorably by
others.
• It can take the form of over-reporting "good
behavior" or under-reporting "bad", or
undesirable behavior.

Common Method Bias
• Common-method variance (CMV) is the
spurious "variance that is attributable to the
measurement method rather than to the
constructs the measures are assumed to
represent

Thank you

Construct
• a construct is the abstract idea, underlying
theme, or subject matter that one wishes to
measure

Further reading
• DeVellis, R. F. (2011). Scale development: Theory and applications (Vol. 26).
Sage Publications.
• Polit, D. F., & Beck, C. T. (2006). The content validity index: are you sure
you know what's being reported? Critique and recommendations.
Research in nursing & health, 29(5), 489-497.
• Hinkin, T. R. (1995). A review of scale development practices in the study
of organizations. Journal of management, 21(5), 967-988.
• Hair, J. F., Tatham, R. L., Anderson, R. E., & Black, W. (2006). Multivariate
data analysis (Vol. 6). Upper Saddle River, NJ: Pearson Prentice Hall

Scale development

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scale development

Similar to Scale development (20)

Recently uploaded

Recently uploaded (20)

Scale development