Quantitative Research: Surveys and Experiments

IS541, Lecture 6a: “Quantitative Research: Surveys and Experiments”
Master Management, Master Business Informatics; March 4th, 2015
Martin Kretzer
Chair of Information Systems IV, Business School and
Institute for Enterprise Systems (InES), University of Mannheim

Overall Course Structure
#9Final
Assignment
#5a Literature Review Intro
#5b Literature Review RBD
2ManTIS FSS 2015 - Quantitative
Research
#7a Design Science Intro
#7b Design Science RBD
#8a Qualitative Research Intro
#8b Qualitative Research RBD
#6a Quantitative Research Intro
#6b Quantitative Research RBD
#1 Introduction
#2 Theories
#3 Methods
#4 Scientific Writing
and Publishing

Goals of this Lecture
A
 Understand the basics of survey-
based research and experiments
 Know the research process of
surveys and experiments
 Data gathering and analysis
 Be aware of important quality
criteria for quantitative research
 Learn best practices
 Get to know popular software tools
for analyzing quantitative data
Research

Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
4ManTIS FSS 2015 - Quantitative Research

The Two Main Paradigms of Empirical Work
Quantitative Qualitative
Mostly deductive (theory first)  Mostly deductive (observation first)
Statistical generalizability  Analytical generalizability
Linear, pre-planned research design  Evolving, iterative research design
High number of observations  Focused number of observations
Statistical analyses  Conceptual analyses
Independent of context  Context-dependent
Reliability is key  Authenticity is key
Source: Denzin and Lincoln (2011), Neumann (2000)
After spring breakToday
Research

Characteristics of Quantitative Research
 Strongly connected to a positivist epistemological stance ( lecture 2)
– Objective reality can be captured and translated into testable hypotheses
– Researcher can capture empirical data that allows them to make inferences about that reality
 Generally emphasizes high “n”
– Numbers represent values and levels of theoretical constructs and concepts
– Interpretation of the numbers is viewed as strong scientific evidence of how a phenomenon works
– Aims for statistical generalizability to make predictions among unobserved members of the
population
 Strongly relies on statistical tools as an essential element in the researcher's toolkit
6
Source: Straub et al. (2005)
ManTIS FSS 2015 - Quantitative
Research

Quantitative Research Methods (revisited)
Simulation Field Experiment / Quasi Experiment
 State a hypothesis
 Imitate some real process or action to prove your hypothesis
 Suitable for observing correlation between variables
 Strengths: allows estimation and prediction
 Conducted in field settings, e.g. real organization
 Rare, because of the difficulties associated with manipulating
treatments and controlling for extraneous effects in a field
setting
 Strengths: high internal and external validity
Survey Laboratory Experiment
 Collect self reported data of people
 Standardized questionnaire or interview
 Suited to study preferences, thoughts, and behavior of
people
 Suited for descriptive, exploratory or explanatory research
 Strengths: collect unobservable data (thoughts); remote
collection; convenient for respondents; can detect small
effects
 Independent variables are manipulated by the researcher (as
treatments)
 Subjects are randomly assigned to treatment
 Results of the treatments are observed
 Suited for explanatory research to examine individual cause-
effect relationships in detail
 Strengths: influence of individual factors can be well
explained; very high internal validity (causality)
Source: Bhattacherjee (2012), Myers (2009)
Research

Examples for Quantitative Studies
8
 Survey
– Goal: Identify drivers for user satisfaction
– Build a questionnaire for (1) measuring satisfaction with a new IS and (2) measuring the constructs that
you expect could influence satisfaction.
– Distribute the questionnaire to your sample (e.g. people in your organization)
– Analyze answered questionnaires by analyzing which drivers can be associated with satisfaction (e.g.,
through computing correlation)
 Experiment
– Goal: Examine the effects of your new online shop
– Build your treatment groups, e.g.:
• Online shop that recommends items
• Online shop that does not recommend items
– Randomly assign individuals to a treatment group
– After 4 weeks check whether your recommender increased sales
 Simulation
– Goal: Examine the effect of a new algorithm
– Build your treatment groups (e.g., old algorithm, new algorithm) and check for differences
– Can be referred to as „computational experiment“
Research

conceptual
observable
unobservable
Theory’s Basic Constituents and Mechanisms
Source: Bhattacherjee (2012, p. 39)
Empirical
Plane
Theoretical
Plane Construct A Construct BProposition
Independent
Variable
Dependent
Variable
Hypothesis
External validity
Internal validity
Research

Classes of Variables
10
Intelligence
Academic
Achievement
Earning
Potential
Effort
Independent
Variable
Moderating
Variable
Mediating
Variable
Dependent
Variable
Research

Internal Validity and External Validity
Internal validity External validity
 = Causality
 Does a change in X really cause a change in Y?
 Three conditions
1. Covariation of cause and effect
2. Temporal precedence
3. No plausible alternative explanation
 Note: causality <> correlation!
 = Generalizability
 Can the observed association be generalized from
sample to the population or further contexts?
Research

Internal Validity and External Validity
12
•There is no single „best“ research method!
•You need to know the strengths and limitations of your research method!
•Combinations might make sense (also mixing quantitative and qualitative research
methods; see Venkatesh et al. 2013 for further information)
Research

Internal Validity (Causality)
 Why is internal validity of cross-sectional field surveys typically limited?
– Independent variable Y cannot be manipulated
– Cause and effect are measured at the same time  you do not know whether X
causes Y or Y causes X
 Why is internal validity of laboratory experiments typically high?
– Independent variable Y can be manipulated via a treatment
– Effect can be observed after a certain point in time
– External factors can be controlled
Research

External Validity (Generalizability)
 Why is external validity of laboratory experiments typically limited?
– Artifical treatments
– External factors are controlled  But: in real settings external factors cannot
be controlled!
 Why is external validity of cross-sectional surveys typically high?
– Data from a wide variety of individuals or firms is collected
 Qualitative research: Why may single case studies have higher
generalizability than multiple case studies?
– In qualitative research studies, you have to clearly describe the context of your
study
– The more detailed you can describe the context, the better you can explain to
which further cases your results can be generalized!
Research

conceptual
observable
unobservable
Theory’s Basic Constituents and Mechanisms (cont’d)
Empirical
Plane
Theoretical
Plane Construct A Construct BProposition
Independent
Variable
Dependent
Variable
Hypothesis
External validity
• Internal validity
• Statistical conclusion validity
Construct
validity
Construct
validity
Research

Construct validity and statistical conclusion validity
Construct validity Statistical conclusion validity
 How well is our measurement scale measuring the
independent variable and the theoretical construct
that it is expected to measure?
 Are the statistical conclusions really valid?
 Did we select the right statistical method for testing
the hypothesis?
 Does the sample meet the requirements?
 (only relevant for quantitative research)
Research

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

Using surveys: advantages and disadvantages
Biases
 Non-response bias
 Common method bias
 Sampling bias
 Social desirability bias
Flexible and Efficient High Volume Data Analysis
 Application across all research
phases
 Measure unobservable data
(Preferences, Attitudes, Traits)
 Economical in terms of
researcher time, effort and cost
 Can be administered to a
high number of subjects
 Remote data collection
 Comparability through
standardization
 Well established quality
criteria
 Statistical tests
 Detect small effects with
large samples
Richness of Data
 Only answers to standardized questions
 Interpretation and context of respondent
missing
Source: Bhattacherjee (2012)
Research

Structural and measurement model
19
 Structural model defines abstract relationship between constructs
 Measurement model contains empirically observed variables
Legend:
 latent exogenous variable
 latent endogenous variable
x measurable exogenous indicator
y measurable endogenous indicator
 path coefficient between latent exogenous and
endogenous variables
 path coefficient between latent variables and
measurable indicators
 measurement error
 
y2y1
3
Measurement Model (Outer Model)
Structural Model (Inner Model)


3 4
4
x2x1
1
1 2
2
Source: Williams et al. (2009)
Research

Attention: List of terms that are frequently used
interchangeably in positivistic reseach!
 Measurable indicator (e.g., x1, x2, x3, y1, y2, y3)
– „Question“
– Indicator
– Item
– Measure
– Measurable variable  you should not use this term!
 Independent variable (IV) (e.g., x)
– (Latent) Exogenous variable
– Factor (typically refers to an IV that uses a nominal scale in experiments)
– Antecedent
– Cause
 Dependent variable (DV) (e.g., y)
– (Latent) Endogenous variable
– Outcome
– Effect
 Moderation effect
– Interaction effect
Research

Scale development for measurement indicators
21
Definition Scaling Process:
 Process of developing scales to measure indicators (items)
 Rating scales are attached to items (unidimensional or multidimensional scaling)
 Scales define what the respondent can choose to answer
 Depending on the chosen scale certain statistic analysis are possible
Description Example
EqualAppearing
 Participants rate items with
“agree” or “disagree”
 Items only “appear” equal; in fact they represent
different values for measuring a certain concept
Thurstone
agree disagree
I like doing sports. X
I like swimming. X
Summative/Cumulative
Likert
 Participants rate items on a 5-point or
7-point scale
 Scale ranges from “strongly agree” to “strongly
disagree”
strongly strongly
agree … neutral … disagree
I like …. X
I like…. X
Guttman
 Goal: Cumulative scale
 Participants rate items with “yes”/“no”
 Creates a sorted matrix or table (see example)
yes no
Do you mind immigrants in your city? X
Would you live next to an immigrant? X
Would you marry an immigrant? X
Source: Bhattacherjee (2012, pp. 48-50)
ManTIS FSS 2015 - Quantitative Research

Number of Items per Construct
 Why is the number of items for measuring a variable so important?
– Problem with only one item: risk of random error may be high
– Problem with too many items: people will stop answering your survey
 How many questions ( items) should I ask?
–  Almost no risk for random erorr: 1 item
–  Stable constructs: 3-6 items
–  Rather new constructs: at least 8 items
 Examples:
– Age  1 item
– Perceived ease of use (Davis 1989):
• First pretest: 16 items
• Final items: 6 items
– Perceived ease of use (today): usually 3 items
Research

Quality Criteria: Measurement Model Validity
Summary of quality criteria for the measurement model
Research
Quality criterion Recommendation
Convergent validity of a
single indicator
High average factor loadings: λ > 0.7
Narrow range of factor loadings: λmax - λmin < 0.2
single construct
High average variance extracted: AVE > 0.5
High composite reliability: ρc > 0.8
High communality index of the construct
Convergent validity of the
measurement model
High average communality index
Discriminant validity of a
single indicator
Each item loading is greater than all cross-loadings
single construct
A construct’s AVE is greater than the squared construct’s
correlation with any other construct
No common method bias
Substantive factor loadings are greater than method factor
loadings
Method factor loadings are not significant while
substantive factor loadings are significant
(Details in section 6 “Supplementary Material on Quality Criteria”)

Quality Criteria: Structural Model Validity
Research
Summary of quality criteria for the structural model (PLS regression analysis)
Direct effect
High standardized path estimates
Bootstrap algorithm and t-test
Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS
Predicting power
High variance explained (R² > 0.2)
High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect
High redundancy
Global quality of structural
model
High goodness of fit
No multicollinearity
No perfect correlation between independent variables: Standardized path
estimates < 0.8
High tolerance of independent variables: tolerance > 0.1
Small variance inflation factor of independent variables: VIF < 10
(Details in section 6 “Supplementary Material on Quality Criteria”)

Golden Circle Analysis – Survey Example
 Golden Circle Analysis (GCA)
 “Privacy Concerns and Privacy-Protective
Behavior in Synchronous Online Social
Interactions”
– Authors: Z.J. Jiang, C.S. Heng, B.C.F. Choi
– Year: 2013
– Outlet: Information Systems Research (ISR)
• Vol. 24, No. 3, pp. 579-595
Research
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis

GCA Survey: Brief Summary
 Individuals‘ behavior is at times inconsistent with their privacy
concerns (e.g., they disclose private information in synchronous
online social interaction although they know the risks)
 Focus: privacy concerns versus social rewards
 Students conduct 3 chatroom sessions  afterwards a survey is
administered
 Findings:
– Individuals use self-disclosure and misrepresentation to protect their privacy
– Social rewards explain deviations
26
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Motivation
 Privacy trade-off in the context of online social interactions
– Online social interactions may generatie multiple benefits:
synchronous exchange of information, sharing of cultural artifacts,
self-presentation, feedback from peers, socio-emotional support,
a borderless „space“
– However, 33% of internet users are concerned about their privacy
in online social interactions ( the paper lists numerous threats)
– Ironically, many users are still likely to disclose private information
even if they become aware of the risks
27
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Research Question
28
 The paper clearly states a research question:
– Why is users‘ privacy behavior at times inconsistent with their
privacy concerns?
– P. 580, end of second paragraph
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Research Objective
29
 The paper provides a clear definition of its research
objectives
– P. 580, last paragraph
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Theory Base
 Paper is not based on a single, specific theory
 However, the paper integrates two research streams to build a research model:
– Hyperpersonal framework
• Approach for understanding how users of mediated communications experience relational
intimacy
– Privacy calculus
• Relational privacy trade-off: privacy concerns versus rewards from disclosing private
information
 Research Model:
30
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Research Design – Data Collection
 Pilot: 3 rounds of preliminary tests to compare and evaluate different
methods of data collection (Appendix A)
 Research Design
– Sample: 251 students in Singapore
– Three online chat sessions, each lasting 1 hour
 Survey
– Survey after the end of the third chat session
– All items measured on a 7-point Likert scale ranging from 1 „strongly
disagree“ to 7 „strongly agree“. E.g., privacy concerns awareness:
31
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Research Design – Data Analysis
 Data Analysis:
– Partial least squares (PLS) regression
– Measurement model assessment:
• Individual item reliability
• Internal consistency
• Discriminant validity
• Item loadings, cross-loadings, composite reliability, average variance
extracted (AVE)
– Structural model assessment
• Correlations
• Path coefficients and hypotheses testing  all hypotheses were confirmed
• R²
• Sobel tests to examine whether privacy concerns and social rewards fully
mediate the effects
• Confirmatory factor analysis (CFA) for two models in order to test for
common method bias
32
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 Measurement model assessment:
33
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 Measurement model and structural model assessment:
34
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 Structural model assessment:
35
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Contribution
 Extension of the privacy calculus perspective to the context of
synchronous online social interactions
– This contribution is valuable, because past research „has
predominantly applied the privacy calculus to commercial contexts“ (p.
590)
– „In the absence of monetary or tangible rewards, social rewards are
just as attractive in balancing privacy concerns and governing
individuals‘ behavior.“ (p. 590)
 Identification of four antecedents (hyperpersonal framework) of
privacy concerns and social rewards
 Disclosure and nondisclosre are not the only two possible actions
stemming from privacy protection  misrepresentation is a third
action (and independent from the established two actions)
 Explanation of the different roles „anonymity of self“ and
„anonymity of others“ in online social interactions
36
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Analysis of Links
 Practical motivation results in the research question (RQ)
 Theoretical motivation builds on RQ and results in three
clearly stated research objectives
 To address the research objectives, two research streams
are reviewed and integrated into one model
 A survey for confirming the model seems to be an excellent
choice
 The contribution section (section 6.2) summarizes the survey
results and explains how they extend existing literature.
Thereby, it directly addresses the RQ and the motivation of
the paper ( “golden circle”)
37
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Survey: Summary of Analysis
 All elements of the GCA are addressed in the paper
 The reader can easily follow the central theme („Roter
Faden“)
 Contribution seems valuable
 10 Hypotheses are confirmed and typical quality criteria for
surveys is met
 Limitations and future research directions are also oulined
38
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

Experiment Designs
 Between-subjects design
– = Participants can be part of only one treatment group (and are then compared to the „control
group“)
– Advantage: no carryover effects
 Within-subjects design
– = Every single participant is subject to each treatment (incl. The control)
– Advantage: statistical significance
 Mixed design
– E.g., between-subjects design for independent variable A and within-subjects design for
independent variable B
– Example: Mixed experiment design of Master thesis on the next few slides
Research

Mixed Design Experiment Example: Model
41
Perceived ease of use
Expertise
Factor 1
Business Intelligence
Client
Factor 2
Report Recommendation
Factor 3
Potential interaction effects:
• Factor 1 * Factor 2
• Factor 1 * Factor 2 * Factor 3
•  7 effects that should be tested!
Design:
• Factor 1 (Expertise): within-subjects variable
• Factor 2 (BI Client): within-subjects variable
• Factor 3 (Rep. Rec.): between subjects variable
Research

Mixed Design Experiment Example: Design
Introduction
User Expertise Ease of Use of
SAP
BusinessObjects
Ease of Use of
Microsoft Excel
Ease of Use of
Tableau Desktop
Perceived ease of use
of recommendation
Counterbalanced
Counterbalanced
Factor 2: Business Intelligence Client
Factor 3: Recommender
Factor 1:
User Expertise
Research

Mixed Design Experiment Example: ANOVA Results
 Statistical analysis of multiple factors (i.e., nominally scaled independent
variables) on one independent variable  mostly Analysis of Variance
(ANOVA )
 If you have multiple independent variables  Multiple ANOVA
(MANOVA)
43
Df Sum Sq Mean Sq F value P(>F)
Between-subjects:
EXP 4 0.42 0.106 0.048 0.995
CLIENT 2 6.93 3.464 1.587 0.227
EXP*CLIENT 8 24.23 3.029 1.387 0.256
Residuals 22 48.03 2.183
Within-subjects:
REC 1 4.879 4.879 4.010 0.058+
REC*EXP 4 14.629 3.657 3.006 0.040*
REC*CLIENT 2 0.496 0.248 0.204 0.817
REC*EXP*CLIENT 8 13.952 1.744 1.433 0.238
Residuals 22 26.769 1.217
Dependent variable: perceived ease of use; n=37. Significance: *p<0.05; +p<0.10
Research

Mixed Design Experiment Example: Graph. Results
44
 (M)ANOVA only indicates effects but no directions of the effect!
 Thus, you need to draw the effect and interpret the figure!
 Note: If the lines in your graphic are parallel, then there is no interaction effect at all!
Research

Common ways for increasing internal validity of experiments
• Manipulate independent variables in one or more levels (treatment)
• Compare the effects of the treatments against a control group
• In experimental designs subjects must recognize different treatments
Treatment
• Eliminate extraneous variables by holding them constant
• For example restricting a study to a single gender
Elimination
• Consider additional extraneous variables
• Separately estimate their effects on the dependent variable
(e.g., via factorial designs where one factor is gender)
Inclusion
• Measure extraneous variables
• Use them as covariates during the statistical testing process
Statistical control
• Cancel out effects of extraneous variables through a process of random sampling (if random nature is
proven)
• Two types: random selection, random assignment
Randomization
• Randomize the order of experimental treatments
• Reduces error due to carryover effectsCounterbalance
45
Source: Bhattacherjee (2012, pp. 39-40)
Research

Golden Circle Analysis – Experiment Example
 Golden Circle Analysis (GCA)
 “The Nature and Consequences of Trade-Off
Transparency in the Context of
Recommendation Agents”
– Authors: J.D. Xu, I. Benbasat, R.T. Confetelli
– Year: 2014
– Outlet: Management Information Systems Quarterly
(MISQ)
• Vol. 38, No. 2, pp. 379-406
Research
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis

GCA Experiment: Brief Summary
 The authors investigate the impact of a novel design
feature for a recommendation agent (RA): trade-off
transparency (TOT)
 The TOT design feature directly influences „consumer‘s perceived
product diagnosticity“ and „perceived enjoyment“
 The authors find that there exists an optimal maximum in TOT
 Furthermore, the authors identify diagnosticity and enjoyment as two
antecedents for „perceived decision quality“ and „perceived decision
effort“
47
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Motivation
 The authors reference four sources which indicate the
economic necessity of RAs for online shops.  But: poorly
designed RAs have negative effects!
 Influence of specific design attributes of RAs on decision
making and other outcomes is still not well understood
 Overall many sources that indicate benefit and importance
of RAs
 RAs ability to capture consumer‘s product attribute
preferences is identified as a „central function of RAs“
 Explanation of potential benefits that might arise if users
have better knowledge about pros and cons of different
laptops when browsing an online shop
48
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Motivation
 Explicit identification of a research gap that needs to be
filled:
– P. 380, last sentence of third paragraph
49
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Research Question
50
 No specific research question statedBrief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Research Objective
51
 Three research objectives are stated specifically
1. Examine the impact of a trade-off transparent RA on perceived
enjoyment and perceived product diagnosticity. The context are
laptops in an online shop.  Example trade-off: price vs. hard-drive
capacity; weight vs. screen size
2. Examine whether there is an optimal maximum of TOT
3. Extend and challenge the effort-accuracy framework because the RA
enables more accurate decisions to be made without simultaneously
increasing efforts
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Theory Base
 Stimulus-Organism-Response (S-O-R) model
– Adopted from marketing and psychology research
– External cues (e.g., design features of online shops) influence a consumer‘s affective and/or
cognitive processes; which in turn determine the consumer‘s behavioral and/or internal
response
– Overarching framework for the authors‘ own theoretical model  Operationalization:
• Stimulus: trade-off transparency feature of an online RA
• Organism: the user‘s enjoyment (affective system), perceived product diagnosticity
(cognitive system)
• Response: the user‘s perceived decision quality and perceived decision effort
 Cognitive load theory
– TOT improves decisions up to a certian point. After that point, TOT overburdens users‘
cognitive limitations and is counterproductive
 Effort-Accuracy framework
– An increase in decision accuracy is accompanied by an increasing in the decision makers
efforts (the „longstanding effort-accuracy conflict“)
52
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Theory Base
 Proposed Theoretical Model
– Enjoyment  an affective reaction (i.e., an emotional response when interacting with a stimulus)
– Product diagnosticity = extent to which a consumer believes that a system is helpful for fully
evaluating a product  a cognitive reaction (i.e., a user‘s mental process when interacting with
the stimulus)
 Based on their proposed theoretical model, the authors develop 10 hypotheses
– Note: The authors hypothesize inverted U-shaped curves as the level of trade-off increases on
perceived enjoyment (H3) and perceived product diagnosticity (H4) --> assumption: an optimal
maximum exists!
– Research model on next few slides
53
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Research Design – Data Collection
 Implementation of design feature
– Horizontal scales with „slider“ represent the value of each product attribute
– If the user moves one slider, other sliders will automatically be moved, too  the
user can directly observe the trade-off dependencies between several attributes
– The greater the TOT, the more sliders are moved automatically
– In total, the online shop offers 50 laptops
54
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 160 participants (of which 131 are undergraduate students)
 Treatment groups:
– „Self-Shoppers“ versus „Friend Shoppers“
• 50% of participants are asked to shop for themselves: if participants are
shopping for themselves, their initial product preference can be compared with
their final attribute preferences
• 50% of participants are asked to shop for a fictitious fried: prior research
indicates that shopping for friends helps minimize the effects of negative
emotions when making attribute trade-offs
– Trade-off transparency
• Low (25% of participants)
• Medium (25% of participants)
• High (25% of participants)
• Control  no specific trade-off transparency (25% of participants)
 Between-subjects design
– Each participant is assigned to one of the 2*4 = 8 treatment groups
– 160 / 8 = 20 participants per group  a power analysis test indicates
sufficient statistical power (0.80) to detect a medium effect size (f=0.25)
55
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 Instructions for participants
 Experimental procedure
– Questionnaire related to demographic and control variables  website training 
random assignment to a group  Questionnaire related to DVs
56
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research
Instruction for “Self-Shoppers” Instruction for “Friend Shoppers”
 After the instruction, the user selects a
value range (in USD) for eight laptop
attributes

GCA Experiment: Research Design – Data Analysis
 Statistical analysis
– MANOVA for testing the effects of trade-off transparency
– PLS regression for testing
• All items from previous literature (p. 391, tbl. 4)
57
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 Manipulation check
– Numbers of shown trade-offs was measured
– Users‘ awareness of trade-offs was measured
 Effect of Trade-Off Transparency levels
– MANOVA (incl. Pillari‘s trace, Wilk‘s lambda, Hotelling‘s trace, Roy‘s largest
root)  results are significant  further ANOVAs on the two DV‘s separately
– Product diagnosticity
• s
58
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

 PLS results
– Measurement model assessment: loadings and cross loadings
– Structural model assessment: composite reliability, Cronbach‘s alpha, AVE,
path coefficients, R²
– Since this GCA focuses on the paper as an exemplary experiment paper, the
PLS results of the questionnaire are not presented in detail  for PLS
analysis, please have a look at the survey GCA
59
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Contribution
 Results
– Trade-off transparent RA improves perceived enjoyment and perceived product
diagnosticity
– Medium level of TOT has the best effect
– Besides H8, all hypotheses are confirmed
 TOT feature helps to identify how users‘ attribute choices are related to,
and are constrained by, one another
 Prior research proposed that trade-off awareness creates unfavorable
feelings (p. 400; Luce et al., 1999)
– But: the authors show that the TOT feature creates positive emotions!
– Reasons: additional content is conveyed (i.e., relationship among product
attribute values) and the interactive presentation
 Contribution to task complexity literature (in particular coordinative
complexity as one dimension of task complexity) by analyzing how the
different number of revealed trade-off relationships influencesn users‘
evaluations
 Both enjoyment and product diagnosticity improve perceived decision
quality without increasing perceived decision effort
 In addition, some practical contributions are derived
60
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Analysis of Links
 The motivation identifies a research gap which is directly addressed by the
research objectives
 Three theory bases are referenced and integrated in order to develop and
propose a model that adresses the three research objectives
 To investigate the proposed model, the authors select a confirmatory research
design. In particular, they select a combination of an experiment and a survey.
The impact of the TOT feature is tested using a between-subjects experiment
design. Further effects are tested using a survey design.
 The contributions section directly builds on the analysis of the experiment (and
the survey)
 Furthermore, the contributions link back to the motivation by answering the three
identified research objectives
61
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

GCA Experiment: Summary of Analysis
 Although a research question is not explicitly stated, the
reader can easily follow the authors‘ work
 GCA elements are addressed and links between them are
straightforward
 Quality criteria for experiments (e.g., manipulation check)
and surveys (e.g., item loadings) are reported
 9 of 10 hypotheses are confirmed and the rejected
hypotheses intuitevely seems to be true in a real setting, too
 Limitations are outlined (e.g., students as subjects, little or
no experience with the RA, laptops is a very customizable
product)
 Overall, the inferences drawn appear to be valid
62
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
Research

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

Software Tool Presentation
 Live Demo
– Survey  PLS regression in software „SmartPLS Version 2“
– Experiment  MANOVA in software „R Studio“
Research

Recommended Materials
 PLS algorithm and SmartPLS software:
– Hair, J.F., Hult, T.M., Ringle, C.M., Sarstedt, M. 2013. A Primer on Partial Least
Squares Structural Equation Modeling (PLS-SEM), Sage Publications.
– https://www.youtube.com/user/Gaskination/playlists
 Statistical programming language R:
– Just search for it using Google and/or YouTube
Research

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

Today‘s Lecture in Review
67
 You learned about the foundations of
quantitative, survey-based research
 You have some advice on study design
 You know about the fundamental process
of study design and execution
 You are familiar with the most important
steps for validating your study
 You know the basic quality criteria and
strategies to ensure them ( more details
in supplementary slides!)
 You have the basic tools for the discussion
of the survey-based papers
 You have seen popular software tools for
conducting quantitative research
Research

What did we exclude?
 Sampling
– In IS important: Who are you talking to? Users? Developers? What is your subject‘s expertise? How often
are they using the IS of interest?...
–  E.g. surveying SAP employees about ERP software would probably cause a huge error
 Scale development
– In case you need to examine new items in your thesis, your supervisor will explain you how to do this 
we assumed all questions can be taken from previous literature
– Formative versus reflective measures  as long as you can take your questions from preivous literature,
this should not bother you too much
 Statistical analyses: Regression, PLS/CB-SEM, (M)AN(C)OVA…
– There are multiple specialization courses offered at the University that you can take for this
– Your thesis supervisor can help you in choosing an appropriate statistical method
Research

Questions, Comments, Observations 6
9
Research

Homework until Next Week
 Write a Golden Circle Analysis (3 pages text) for one of the following papers
and be able to present and discuss your analysis:
–Survey:
• Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-
Acceptance Information System Usage Behaviors: An Investigation in the Business
Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682
• Note: supplementary material in additional pdf-file!
–Field experiment:
• Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering
Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“
Academy of Management Journal (56:5), pp1372-1395
–Experiment + Survey:
• Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use
of Technology,“ MIS Quarterly (37:4), pp. 1013-1041
Research

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

Quality Criteria: Intro to Quality Criteria
 Constructs (theoretical level)
– Constructs are imaginary creations in
our minds
– Definitions or constructs are not
objective, but shared (“inter-
subjective”) agreements
 Two forms of constructs
– Unidimensional constructs have a
single underlying dimension
– Multidimensional constructs consist of
two or more underlying dimensions
 Unobservable theoretical constructs
are translated into indicators
 Indicators are questions that can be
empirically observed and measured
 Example: socioeconomic status is
measured by asking for
– Family income
– Education
– Occupation
 Can be measured multidimensional
or unidimensional
Definition Conceptualization
 Mental process translating imprecise
concepts into precise definitions
 Understand and define what is included
and excluded in a concept
Definition Operationalization
 Process of developing indicators to
measure abstract constructs
 Is based on conceptualization
72
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Quality Criteria: Intro to Quality Criteria
 Three major types of validity in quantitative research (Cook and Campbell 1979; Shadish et al. 2002):
– Design validity
– Measurement validity
– Inferential validity
 Design validity refers to internal validity (causality and control for alternative explanations) and
external validity (generalizability)
– See previous slides
– E.g., students as participants  valid design?  potential issues are usually mentioned in the limitation section
 Measurement validity estimates how well items measure what they are purported to measure
according to their definitions
 Inferential validity, also called statistical conclusion validity, refers to the correct application of
statistical procedures to find relationships.
 Note: This summary of quality criteria focuses on PLS-SEM.
– PLS is a prediction-oriented variance-based approach that focuses on endogenous target constructs in the
model and aims at maximizing their explained variance, i.e., their R² value (Hair et al. 2012a).
– PLS has become a quasi-standard (e.g., Bagozzi and Yi 2012; Hair et al. 2012b; Ringle et al. 2012; Shook et al.
2004; Steenkamp and Baumgartner 2000)
– I do not provide a detailed comparison of the two approaches, because it would go beyond the scope of this
presentation (e.g., Chin and Newsted 1999; Chin et al. 2003; Marcoulides et al. 2009; Qureshi and Compeau
2009) and there is still an ongoing debate about strengths and weaknesses of the two approaches (Goodhue et
al. 2012a, 2012b; Marcoulides et al. 2012)
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

 Two quality goals (Bagozzi and Yi 2012):
– Discriminant validity
– Convergent validity
 Discriminant validity refers to the degree to which a construct is more strongly related
to its own indicators than with any other construct (Chin 2010)
– Discriminant validity assures that indicators are assigned to the correct construct
and multiple constructs do not overlap in their definitions
 Convergent validity refers to the degree to which a block of items – usually all
indicators of a specific construct – agree (i.e., converge) in their representation of the
construct they are supposed to measure (Chin 2010)
– Convergent validity assures that a set of indicators measures the same construct
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Convergent validity (1/4)
 A quasi-standard is reporting factor loadings to show that various indicators are
measuring the same construct.
 If loadings would be mixed and have a wide range (e.g., varying from 0.5 to 0.9), this
would raise concern about whether the indicators are a homogenous set that primarily
captures the phenomenon of interest (Chin 2010).
 Literature argues that all indicators should be significant, exceed 0.7, and the
difference between indicators measuring the same construct should not exceed 0.2
(Bagozzi and Yi 2012; Chin 2010; Fornell and Larcker 1981).
 Similarly, on the construct level, the shared variance of a set of indicators in relation to
their shared variance plus measurement errors, attempts to measure the amount of
variance that a construct extracts from its indicators, so-called average variance
extracted (AVE). It is computed as
where 𝜆𝑖 is the factor loading connecting an indicator to its hypothesized factor and
𝜃𝑖𝑖 is the variance of the error term corresponding to the indicator (Bagozzi and Yi
2012; Fornell and Larcker 1981).
Research
 
  iii
i
iindicator
factor
factor
AVE







var
var
2
2
_
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

 At least 50% variance of items should be accounted for, leading to a minimum AVE of
0.5 (Bagozzi and Yi 2012; Chin 2010; Hair et al. 2014).
 Although many researchers have compared the square root of AVE to construct
correlations, you can equivalently compare the AVE to squared correlations among
constructs as this has two advantages (Chin 2010):
– The shared variance is represented in terms of percentage overlap
– Differences are easier to distinguish
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

 Another index measuring a construct’s reliability based on convergent validity of its
indicators, is the composite reliability index 𝜌 𝐶, given by
where 𝜆𝑖𝑗 refers to factor loading i on factor j and 𝜃𝑖𝑖 is the variance of the error term
corresponding to the indicator (Bagozzi and Yi 2012; Werts et al. 1974)
 In contrast to Cronbach’s alpha, which is a minimum estimate of reliability, composite
reliability does not assume that all items are equally weighted and thus can be a better
estimate of reliability. Like AVE, 𝜌 𝐶 is only applicable for constructs measured with
reflective indicators, too. According to recommendations in literature, 𝜌 𝐶 should be at
least greater than 0.6 for new constructs (Hair et al. 2014) and greater than 0.8 for
stable constructs (Fornell and Larcker 1981).
Research
   
    



iiij
ij
compositeC
factor
factor



var
var
2
2
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

 Furthermore, the average variance an indicator explains is measured through the
communality index and is computed as
where pq is the amount of indicators of the latent variable q and is the
squared factor loading of one indicator of q (Vinzi et al. 2010).
 Regarding the entire measurement model, the average communality index is defined
as
where p is the total number of indicators in the model, J is the amount of latent
variables, and pj is the amount of indicators of J (Vinzi et al. 2010).
 Note: Although the communality index indicates quality of constructs and the average communality index indicates
the quality of the overall measurement model, communality scores are frequently reported together with quality
criteria of the structural model. The reason for this is, that, based on communality, further indices indicating quality of
the structural model can be calculated and, for the reader, data analysis is be easier to understand if communality
and indices based on communality are reported together.
Research
 

qp
p
qpq
q
q xcor
p
com
1
2
,
1

 qpqxcor ,2
 

J
j
jj comp
p
com
1
*
1
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Discriminant validity
 Recent research recommends to prove that a construct is more correlated with its own
indicators than with the indicators of another construct (Chin 2010). Otherwise there
would be the option that multiple constructs share the same types of indicators and
thus would not be conceptually distinct.
 Prior research recommends comparing all constructs’AVE indices with their respective
squared correlations to other constructs – or equivalently all constructs’ square root of
the AVE indices with their respective correlations to other constructs (Chin 2010;
Fornell and Larcker 1981). If a construct’s AVE score is higher than all squared
correlations to other constructs, the construct is more strongly related to its own
indicators than to the indicators of another construct.
 Furthermore, literature recommends comparison of correlations between indicators
and constructs in order to argue for discriminant validity (Chin 2010). That is, loadings
of an indicator to the construct it is supposed to measure (i.e., factor loadings) should
be greater than all loadings of the same indicator to other constructs (i.e., cross
loadings).
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Common method bias (1/2)
 Besides convergent and discriminant validity, bias caused by common method
variance (CMV) can be a potential threat, because the exogenous and endogenous
variables are not obtained from different sources (Podsakoff et al. 2003).
 The effects of an unmeasured latent methods factor are controlled by including a
common method factor in the PLS model whose indicators included all the principal
constructs’ indicators and should not be significant (Liang et al. 2007; Podsakoff et al.
2003; Richardson et al. 2009; Williams et al. 2003).
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Common method bias (2/2)
 Podsakoff et al. (2003) suggest adding a latent method factor to the structural model
which is measured using all indicators. The figure below (adopted from Liang et al.
2007) shows an example with the exogenous variable A and the endogenous variable
B, indicators a1, a2, b1, and b2, measurement errors 𝑒1
𝑎
, 𝑒2
𝑎
, 𝑒1
𝑏
and 𝑒2
𝑏
, and factor
loadings 𝜆1
𝑎
, 𝜆2
𝑎
, 𝜆1
𝑏
and 𝜆2
𝑏
.
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Summary of quality criteria for the measurement model
Research
single indicator
High average factor loadings: λ > 0.7
Narrow range of factor loadings: λmax - λmin < 0.2
single construct
High average variance extracted: AVE > 0.5
High composite reliability: ρc > 0.8
High communality index of the construct
Convergent validity of the
measurement model
High average communality index
single indicator
Each item loading is greater than all cross-loadings
single construct
A construct’s AVE is greater than the squared construct’s
correlation with any other construct
No common method bias
Substantive factor loadings are greater than method factor
loadings
Method factor loadings are not significant while
substantive factor loadings are significant
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Estimation of standardized path estimates (PLS)
 The distribution-free PLS approach estimates standardized path estimates based on
shared variances of the associated constructs
 The significance of these estimates are typically assessed using a nonparametric
bootstrapping algorithm (Chin 1998; Chin 2010)
 This algorithm is based on n samples with m cases each (Efron and Tibshirani 1993).
– First, for each case all indicators are replaced with a value from their confidence
intervals
– Then, based on m values per indicator, a value for the sample is computed
– This procedure continues until n samples are calculated
 The accuracy of the bootstrapping algorithm increases with the amount of cases and
samples on which it is based. Literature recommends using default software properties
for the amount of samples and cases when performing bootstrapping analyzes,
because then research results would be comparable (Temme et al. 2010).
 For instance, the bootstrapping algorithm implemented in the software tool
“SmartPLS” estimates (per default) the significance of standardized paths based on
200 samples with 100 cases each
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Interaction effects
 Note: Broad IS research has predominantly employed multiple regression- and
ANOVA-based analytic techniques to investigate interaction terms
– You do not have to use PLS for testing interaction effects!
 Recent literature suggests to use the product-indicator approach in conjunction with
PLS as described by Chin et al. (2003)
– Advantages:
• This approach requires fewer indicators per construct and a smaller sample
size to find true interaction scores
• Furthermore, it is able to handle measurement error, produce consistent
results, and has a smaller tendency to underestimate paths coefficients
– Disadvantage:
• However, it has a slight tendency to overestimate factor loadings
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Predictive Power (1/2)
 Besides strengths of associations between various constructs, predictive power of the
structural model needs to be quantified.
 Commonly predictive power is reported through R² values of the endogenous
constructs. Falk and Miller (1992) recommend that R² values should be greater than
0.1. Hair et al. (2014) recommend that R² values should be greater than 0.2.
 A change in the R² values can further be explored to see whether a particular variable
has a significant effect on another particular variable (Chin 2010). Specifically, the
effect size 𝑓2
should be calculated:
where 𝑅𝑖𝑛𝑐𝑙𝑢𝑑𝑒𝑑
2
and 𝑅 𝑒𝑥𝑐𝑙𝑢𝑑𝑒𝑑
2
are the R² values provided on the dependent latent
variable when the predicting latent variable is used or omitted in the structural equation
respectively.
 According to Cohen (1988), an effect size 𝑓2 of 0.02, 0.15, and 0.35 can be
interpreted as a small, medium, or large impact.
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
22
2
1 included
excludedincluded
R
RR
f



Design validity

Predictive Power (2/2)
 Besides the R² scores, the redundancy index attempts to measure the quality of the
structural model for an endogenous construct, too (Tenenhaus et al. 2005).
 While the R² scores only consider relationships predicting one endogenous construct,
the redundancy index regards the entire structural model (Vinzi et al. 2010).
 Furthermore, the redundancy index combines (a part of) the quality of the
measurement model (i.e., communality index) with (a part of) the quality of the
structural model (i.e., R² values):
 Likewise, the quality of the overall structural model is expressed by the average
redundancy (Vinzi et al. 2010) computed as
where J is the total number of endogenous latent constructs in the model (Vinzi et al.
2010).
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
jjj Rcomred 


J
j
jred
J
red
1
1
Design validity

Overall (i.e., measurement model + structural model) fit criterion
 Furthermore, a global criterion of goodness of fit (GoF) which takes into account the
model performance in both the measurement and the structural model can be
computed
 GoF provides a single measure for the overall prediction performance of the model
(Tenenhaus et al. 2005; Vinzi et al. 2010)
 GoF is computed as the geometric mean of the average communality and the average
R² value:
 Since PLS does not optimize any global function, there is no index that can provide
the user with a global validation of the model (as it is instead the case with 𝜒² [Chi
Square] and related measures; Tenenhaus et al. 2005)
 However, the GoF index represents an operational solution to this problem as it may
be meant as an index for validating the PLS model globally (Duarte and Raposo 2010)
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
RcomGoF Design validity

Multicollinearity (1/3)
 While the indices introduced on the previous slides favor strong correlations between
independent and dependent variables, very strong correlations between several
independent variables (commonly known as multicollinearity) are undesired, because
for each regression coefficient there would be an infinite number of combinations of
coefficients that would work equally well thus making it impossible to obtain unique
estimates of the regression coefficients (Field et al. 2012)
– In other words, if there are two predictors that are perfectly correlated, then the
regression coefficients for each variable would be interchangeable
– This could lead to reduced statistical power, untrustworthy regression coefficients,
high sensitivity to small changes in the data, and difficulties to assess the individual
importance of independent variables (Field et al. 2012)
 Consequence: correlations between independent variables should be smaller than 0.8
(Field et al. 2012).
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

 Multicollinearity can be detected through the tolerance and variance inflation factor
indices.
 Tolerance is the proportion of variance in an independent variable which is not
predicted by the other independent variables (Clark-Carter 2010).
 In order to calculate a certain independent variable’s tolerance, that variable is treated
as dependent variable with all other independent variables as predictors. The obtained
R² is then used to determine the variable’s tolerance index:
 Similarly, the variance inflation factor (VIF) is computed as
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
1 Rtolerance 
2
1
11
Rtolerance
VIF


Design validity

 Recent literature argues that tolerance should be greater than 0.1, meaning that at
least 10% of an independent variable’s variance should not be explained by other
independent variables yet (Clark-Carter 2010; Meyers et al. 2006)
 Equivalently, VIF should be smaller than 10 (Stevens 2002).
 However, O’Brien (2007) argues that the stability of estimated coefficients can be
influenced by other factors. Hence, the variance of the regression coefficients would
be reduced and VIF values of 40 or more could still be acceptable.
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Summary of quality criteria for the structural model (PLS regression analysis)
Direct effect
High standardized path estimates
Bootstrap algorithm and t-test
Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS
Predicting power
High variance explained (R² > 0.2)
High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect
High redundancy
Global quality of structural
model
High goodness of fit
No multicollinearity
No perfect correlation between independent variables: Standardized path
estimates < 0.8
High tolerance of independent variables: tolerance > 0.1
Small variance inflation factor of independent variables: VIF < 10
Design validity

Quality Criteria: Design Validity
92
• No responses due to a systematic reason
• E.g., dissatisfied customers tend to be more vocal
Non-response bias
• Parts of the population are excluded
• E.g., online surveys exclude people without web
Sampling bias
• Tendency to portray oneself socially desirable
• E.g., “Have you ever downloaded illegal music?”
Social desirability bias
• Participants might not remember certain events
• E.g., “For which tasks have you used your personal
computer ten years ago?”
Recall bias
• Variables measured with an identical method, and
• Variables measured at the same time
Common method bias
Source: Bhattacherjee (2012)
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity

Agenda
Agenda
2 Surveys
3 Experiments
5 Summary
7 References

References (1/4)
94
 Bagozzi, R. P., and Yi, Y. 2012. “Specification, Evaluation, and Interpretation of Structural Equation Models,” Journal of the Academy of Marketing Science 40(8), pp. 8-34.
 Bhattacherjee, A. 2012. Social Science Research: Principles, Methods, and Practices, (2. ed.). Tampa, FL, USA: Open Access Textbooks.
 Chin, W. W. 1998. “The Partial Least Squares Approach to Structural Equation Modeling,” in Modern Methods for Business Research, G. A. Marcoulides (ed.), Mahwah, NJ:
pp. 295-336.
 Chin, W. W., and Newsted, P. R. 1999. “Structural Equation Modeling Analysis with Small Samples Using Partial Least Squares,” in Statistical Strategies for Small Sample
Research, R. Hoyle (ed.), Thousand Oaks, CA: Sage Publications, pp. 307-341.
 Chin, W. W., Marcolin, B. L., and Newsted, P. R. 2003. “A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte
Carlo Simulation Study and an Electronic-Mail Emotion / Adoption Study,” Information Systems Research (14:2), pp. 189-217.
 Chin, W. W. 2010. “How to Write Up and Report PLS Analyses,” in Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J.
Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 655-690.
 Clark-Carter, D. 2010. Quantitative Psychological Research. The Complete Student’s Companion (3rd ed.), Hove, UK: Psychology Press.
 Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Erlbaum.
 Cook, T. D., and Campbell, D. T. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings, Boston, MA: Houghton Mifflin Company.
 Cronbach, L.J., 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16 (3), 297–334.
 Denzin, N. K., and Lincoln, Y. S. 2011. The SAGE Handbook of Qualitative Research, (4. ed.), Thousand Oaks, CA, USA: Sage Publications.
 Duarte, P. A. O. and Raposo, M. L. B. 2010. "A PLS Model to Study Brand Preference: An Application to the Mobile Phone Market," in Handbook of Partial Least Squares.
Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 449-485.
 Efron, B., and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, New York, NY: Chapman & Hall.
 Falk, R. F. and Miller, N. B. 1992: A Primer for Soft Modeling, Ohio: The University of Akron Press.
 Field, A., Miles, J., and Field, Z. 2012. Discovering Statistics Using R, London, UK: Sage Publications.
 Fornell, C., and Larcker D. F. 1981. “Evaluating Structural Equation Models with Unobserved Variables and Measurement Error,” Journal of Marketing Research (18), pp. 39-
50.
 Goodhue, D. L., Lewis, W., and Thompson, R. 2012a. “Comparing PLS to Regression and LISREL: A Response to Marcoulides, Chin, and Saunders,” MIS Quarterly (35:3),
pp. 703-716.
 Goodhue, D. L., Lewis, W., and Thompson, R. 2012b. “Does PLS have Advantages for Small Sample Size or Non-Normal Data?” MIS Quarterly (36:3), pp. 981-1001.
Research

References (2/4)
95
 Goodhue, D., Lewis, W., and Thompson, R. 2006. "PLS, Small Sample Size, and Statistical Power in MIS Research," 39. Annual Hawaii International Conference on System
Sciences (HICSS 2006), Kauai, HI, USA, pp. 202b-202b.
 Guba, E.G., and Lincoln, Y.S. 1994. "Competing Paradigms in Qualitative Research," in: Handbook of Qualitative Research, N.K. Denzin and Y.S. Lincoln (eds.). Thousand
Oaks, CA, USA: Sage, pp. 105-117.
 Hair, J. F., Ringle, C. M., and Sarstedt, M. 2012a. “Partial Least Squares: The Better Approach to Structural Equation Modeling?” Long Range Planning (45), pp. 312-319.
 Hair, J. F., Sarstedt, M., Ringle, C. M., and Mena, J. A. 2012b. “An Assessment of the Use of Partial Least Squares Structural Equation Modeling in Marketing Research,”
Journal of the Academy of Marketing Science (40:3), pp. 414-433.
 Hair, J. F., Hult, T. M., Ringle, C. M., Sarstedt, M. 2013. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM): Sage Publications.
 Jiang, Z. J., Heng, C. S., and Choi, B. C. F. 2013. „Privacy Concerns and Privacy-Protective Behavior in Synchronous Online Social Interactions,“ Information Systems
Research (24:3), pp. 579-595.
 Lewis, B. R., Templeton, G. F., and Byrd, T. A. 2005. “A Methodology for Construct Development in MIS Research,” European Journal of Information Systems (14:4), pp. 388-
400.
 Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-Acceptance Information System Usage Behaviors: An Investigation in the Business
Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682.
 Liang, H., Saraf, N., Hu, Q., and Xue, Y. 2007. “Assimilation of Enterprise Systems: The Effect of Institutional Pressures and the Mediating Role of Top Management,” MIS
Quarterly (31:1), pp. 59-87.
 MacKenzie, S.B., Podsakoff, P.M., and Podsakoff, N.P. 2011. "Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and
Existing Techniques," MIS Quarterly (35:2), pp. 293-A295.
 Marcoulides, G. A., Chin, W. W., and Saunders, C. 2012. “When Imprecise Statistical Statements become Problematic: A Response to Goodhue, Lewis, and Thompson,” MIS
Quarterly (36:3), pp. 717-728.
 Marcoulides, G. A., Chin, W. W., and Saunders, C. 2009. “A Critical Look at Partial Least Squares Modeling,” MIS Quarterly (33:1), pp. 171-175.
 Marcoulides, G.A., and Saunders, C. 2006. "PLS: A Silver Bullet?," MIS Quarterly (30:2), pp. iii-ix.
 Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“
Academy of Management Journal (56:5), pp1372-1395.
 Meyers, L. S., Gamst, G., and Guarino, A. J. 2006. Applied Multivariate Research. Design and Interpretation, Thousand Oaks, CA: Sage Publications.
 Monette, D. R., Sullivan, T. J., and DeJong, C. R 2010. Applied Social Research. A Tool for the Human Services, (8. ed.), Belmont, CA, USA: Cengage Learning.
 Myers, M. D. 2009. Qualitative Research in Business & Management, Illustrated edition: Sage Publications.
Research

References (3/4)
96
 Neuman, W.L. 2000. Social Research Methods: Quantitative and Qualitative Approaches, (4. ed.). Boston, MA, USA: Allyn and Bacon.
 O’Brien, R. M. 2007. “A Caution Regarding Rules of Thumb for Variance Inflation Factors,” Quality and Quantity (41), pp. 673-690.
 Pagano, R. R. 2010. Understanding Statistics in the Behavioral Sciences, (10. ed.), Belmont, CA, USA: Cengage Learning.
 Petter, S., Straub, D. W., and Rai, A. 2007. “Specifying formative Constructs in Information Systems Research,” MIS Quarterly (31:4), pp. 623-656.
 Podsakoff, P. M., MacKenzie, S. B., Lee, Y.-J., and Podsakoff, N. P. 2003. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and
Recommended Remedies,” Journal of Applied Psychology (88:5), pp. 879-903.
 Qureshi, I., and Compeau, D. 2009. “Assessing Between-Group Differences in Information Systems Research: A Comparison of Covariance- and Component-based SEM,”
MIS Quarterly (33:1), pp. 197-214.
 Richardson, H. A., Simmering, M. J., and Sturman, M. C. 2009. “A Tale of Three Perspectives: Examining Post Hoc Statistical Techniques for Detection and Correction of
Common Method Variance,” Organizational Research Methods (12:4), pp. 762-800.
 Ringle, C. M., Sarstedt, M, and Straub, D. W. 2012. “A Critical Look at the Use of PLS-SEM in MIS Quarterly,” MIS Quarterly (36:1), iii-xiv.
 Shadish, W. R., Cook, T. D., and Campbell, D. T. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference, Boston, MA: Houghton-Mifflin.
 Shook, C.L., Ketchen, D.J., Hult, T., and Kacmar, K.M. 2004. “An assessment of the use of structural equation modeling in strategic management research,” Strategic
Management Journal, 25(4), pp. 397-404.
 Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use of Technology,“ MIS Quarterly (37:4), pp. 1013-1041.
 Steenkamp, J.-B., and Baumgartner, H. 2000. “On the Use of Structural Equation Models for Marketing and Modeling,” International Journal of Research in Marketing (17:2/3),
pp. 195-202.
 Stevens, J. P. 2002. Applied Multivariate Statistics for the Social Sciences (4th ed.), Mahwah, NJ: Earlbaum.
 Straub, D.W., Gefen, D., and Boudreau, M.-C. 2005. "Quantitative Research," in: Research in Information Systems: A Handbook for Research Supervisors and Their Students,
D. Avison and J. Pries-Heje (eds.). Amsterdam, The Netherlands: Elsevier, pp. 221-238.
 Temme, D., Kreis, H., and Hildebrandt, L. 2010. “A Comparison of Current PLS Path Modeling Software: Features, Ease-of-Use, and Performance,” in Handbook of Partial
Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 737-756.
 Tenenhaus, M., Vinzi, V. E., Chatelin, Y.-M., and Lauro, C. 2005. “PLS Path Modeling,” Computational Statistics & Data Analysis (48), pp. 159-205.
 Venkatesh, V., Brown, S. A., Bala, H. 2013. Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems,“ MIS
Quarterly (37:1), pp. 21-54.
Research

References (4/4)
97
 Vinzi, V. E., Trinchera, L., and Amato, S. 2010. “PLS Path Modeling: From Foundations to Recent Developments and Open Issues for Model Assessment and Improvement,” in
Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 47-82.
 Werts, C. E., Linn, R. L., and Joereskog, K. G. 1974. “Intraclass Reliability Estimates: Testing Structural Assumptions,” Educational and Psychological Measurement (34), pp.
25-33.
 Williams, L. J., Edwards, J. R., and Vandenberg, R. J. 2003. “Recent Advances in Causal Modeling Methods in Organizational and Management Research,” Journal of
Management (29:6), pp. 903-936.
 Williams, L.J., Vandenberg, R.J., and Edwards, J.R. 2009. "Structural Equation Modeling in Management Research: A Guide for Improved Analysis," The Academy of
Management Annals (3:1), January, pp 543-604.
 Xu, J. D., Benbasat, I., and Cenfetelli, R. T. 2014. „The Nature and Consequences of Trade-Off Transparency in the Context of Recommendation Agents,“ MIS Quarterly
(38:2), pp. 379-406.
Research

Contact
Martin Kretzer
Research Assistant
Consultation hour: per request
E-Mail: kretzer@es.uni-mannheim.de
ManTIS 2015 - Overview and
Registration
98

Quantitative Research: Surveys and Experiments

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Quantitative Research: Surveys and Experiments

Similar to Quantitative Research: Surveys and Experiments (20)

Recently uploaded

Recently uploaded (20)

Quantitative Research: Surveys and Experiments