Building and Evaluating Theories   in Software Engineering

Daniel Méndez
TUM, Germany
www.mendezfe.org
Building and Evaluating Theories  
in Software Engineering
@mendezfe
Software Engineering Research Methods
Dagstuhl, April 2019

Simpliﬁed view on epistemological setting
Principle ways of working
Methods and strategies
Fundamental theories
Philosophy of science

Simpliﬁed view on epistemological setting
Philosophy of science Epistemology
Empiricism
Statistics
Hypothesis testing
Controlled Experimentation
Example!

Setting: Empirical Software Engineering
Typically in scope of
Empirical Software  
Engineering

Setting: Empirical Software Engineering
Theory building  
and evaluation
Methods and  
strategies
… supported by…
Analogy: Theoretical and  
Experimental Physics

Goals of this talk
Theory building  
and evaluation
Methods and  
strategies
… supported by…
Analogy: Theoretical and  
Experimental Physics
Basic understanding of
• Notion of theories in our field
• (Philosophy of science) context of
research methods and strategies
for theory building and evaluation

Ground rule
Whenever you have questions / remarks,
don’t ask , but
share them with the whole group.

Theory Building in Software Engineering
‣Theories in a Nutshell
… where we will brieﬂy talk about the general notion of theories
‣State of Evidence in Software Engineering
… where we will see why theory building is so important to our ﬁeld
‣Research Methods in Software Engineering
… where we will put research methods in a larger (philosophical) picture
‣Qualitative “vs” quantitative research
… where we will discuss different research approaches

What is a theory?
What do you think?

Theories (generally speaking)
Examples:
• Vaccinations lead to autism
• Global warming is invented to harm the industry
• Earth is ﬂat
• …
A theory is a belief that there is a pattern in phenomena.
Further examples: Twitter proﬁles by right-wing populists

Theories (generally speaking)
Examples:
• Vaccinations lead to autism
• Global warming is invented to harm the industry
• Earth is flat
• …
A theory is a belief that there is a pattern in phenomena.
Are these theories scientific?
Further examples: Twitter profiles by right-wing populists
» Speculations based on imagination or opinions that often cannot be refuted  
(i.e. logical fallacies)

Scientiﬁc Theories
1. Tests
• Experiments, simulations, …
• Replications
2. Criticism
• Peer reviews / acceptance in the community
• Corroborations / extensions with further theories
A scientiﬁc theory is a belief that there is a pattern in phenomena while
having survived
1. tests against experiences
2. criticism by critical peers

Scientiﬁc Theories
1. Tests
• Experiments, simulations, …
• Replications
2. Criticism
• Peer reviews / acceptance in the community
• Corroborations / extensions with further theories
A scientiﬁc theory is a belief that there is a pattern in phenomena while
having survived
1. tests against experiences
2. criticism by critical peers
Note: Addresses so-called Demarcation
Problem to distinguishing science from non-
science (as per introduction by K. Popper)

“There is no universally agreed
upon deﬁnition of the concept
of an empirically-based theory
[in Software Engineering], nor is
there any uniform terminology
for describing theories.”
Source: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010.
Approach by characteristics

Scientiﬁc Theories have…
Quality criteria:
• Testability
• Empirical support / (high) level of conﬁdence
• Explanatory power
• Usefulness to researchers and / or practitioners
• …
Adopted from: Sjøberg, D., Dybå, T., Anda, B., Hannay, J. Building Theories in Software Engineering, 2010.
Analytical Explanatory Predictive Explanatory &
Predictive
Scope Descriptions and
conceptualisation,  
including
taxonomies,
classifications, and
ontologies 
- What is?
Identification of
phenomena by
identifying causes,
mechanisms or
reasons 
- Why is?
Prediction of what
will happen in the
future  
- What will happen?
Prediction of what will
happen in the future
and explanation 
- What will happen
and why? 
Further information: https://goo.gl/SQQwxt
A purpose:

Quality criteria:
• Testability
• …
Predictive
including
taxonomies,
ontologies 
- What is?
Identification of
phenomena by
identifying causes,
mechanisms or
reasons 
- Why is?
Prediction of what
will happen in the
future  
- What will happen?
and explanation 
- What will happen
and why? 
A purpose:
Note: Many theories in software engineering are
so-called “design [science] theories”, i.e. scientiﬁc
theories about artefacts in a context.
[Artefact speciﬁcation] X [Context assumptions] → [Effects]

Quality criteria:
• Testability
• …
Predictive
including
taxonomies,
ontologies 
- What is?
Identification of
phenomena by
identifying causes,
mechanisms or
reasons 
- Why is?
Prediction of what
will happen in the
future  
- What will happen?
and explanation 
- What will happen
and why? 
Note: Laws “versus” theories
A law is a purely descriptive theory about
phenomena (without explanations), i.e. an
analytical theory.
A purpose:
Note: Many theories in software engineering are
so-called “design [science] theories”, i.e. scientiﬁc
theories about artefacts in a context.
[Artefact speciﬁcation] X [Context assumptions] → [Effects]

Exemplary framework for describing theories
•Constructs: What are the basic elements? 
(Actors, technologies, activities, system entities,
context factors)
•Propositions: How do the constructs interact?
•Explanations: Why are the propositions as speciﬁed?
•Scope: What is the universe of discourse in which the
theory is applicable?

Exemplary framework for describing theories
•Constructs: What are the basic elements? 
(Actors, technologies, activities, system entities,
context factors)
•Propositions: How do the constructs interact?
•Explanations: Why are the propositions as speciﬁed?
•Scope: What is the universe of discourse in which the
theory is applicable?
Example
Source: Wagner, Mendez Fernandez et al. Status Quo in Requirements Engineering: A Theory and a Global Family of Surveys, TOSEM 2018.
Req Elicitation Technique
Interview
Scenario
Prototyping
Facilitated Meetings
Observation
Req Documentation Technique
Structured req list
Domain/business process model
Use case model
Goal model
Data model
Non‐functional req
Textual
Semi‐formal
Formal
Technology
Req Test Alignment Approach
Req review by tester
Coverage by tests
Acceptance criteria
Test derivation from models
Req Change Approach
Product backlog update
Change requests
Trace management
Impact analysis
Activity
Req Elicitation
Req Documentation
Req Change Management
Req Test Alignment
P 1‐5
P 6‐13
P 14‐20
P 21‐24
Actor
Req Engineer
Test Engineer
Req Standard Application
Practice
Control
Tailoring
Req Eng Process Standard
P 25‐28
Req Standard Defintion
Compliance
Development
Tool support
Quality assurance
Project management
Knowledge transfer
Process complexity
Communication demand
Willigness to change
Possibility of standardisation

Req Improvement Means
Continuous improvement
Strengths/weaknesses
Own business unit/role
Req Eng Improvement
P 29-44
P 45-49
Proposition:
“Structured requirements lists are documented
textually in free form or textually with constraints.”
Explanation:
“Free-form and constraint textual requirements
are sufﬁcient for many contexts such as in agile
projects where they only act as reminders for
further conversations.”

Theories and hypotheses
Empirical Approaches
Theory / Theories
(Tentative) Hypothesis
Falsification /  
Corroboration
Theory (Pattern) 
Building
Hypothesis 
Building
Hypothesis
• “[…] a statement that proposes a possible
explanation to some phenomenon or
event” (L. Given, 2008)
• Grounded in theory, testable and
falsifiable
• Often quantified and written as a
conditional statement
Scientific theory
• “[…] based on hypotheses tested and
verified multiple times by detached
researchers” (J. Bortz and N. Döring,
2003)
If cause/assumption (independent variables)
then (=>) consequence (dependent variables)
Note: We don’t “test theories”, but
their consequences via hypotheses  
(i.e. testable propositions)

From real world phenomena to theories  
(and back) - The Empirical Lifecycle
Empirical Approaches
Theory / Theories
(Tentative) Hypothesis
Falsification /  
Corroboration
Theory (Pattern) 
Building
Units of Analysis
Sampling Frame
Sampling
Hypothesis 
Building
Empirical Inquiries
Induction
Inference of a  
general rule  
from a particular
case/result  
(observation)
Abduction
(Creative) Synthesis of an  
explanatory case from a general rule  
and a particular result (observation)
Deduction
Application of a general rule  
to a particular case,  
inferring a speciﬁc result
Source: Mendez and Passoth. Empirical Software  
Engineering: from Discipline to Interdiscipline, 2018.

Example: Goal-oriented RE (“GORE”)
Papers published [1]: 966
[2] Mavin, et al. Does Goal-Oriented Requirements Engineering Achieve its Goal?, 2017
[3] Mendez et al. Naming the Pain in Requirements Engineering Initiative - www.re-survey.org
[1] Horkoff et al. Goal-Oriented Requirements Engineering: A Systematic Literature Map, 2016
Papers including a case study [1]: 131
Studies involving practitioners [2]: 20
Practitioners actually using GORE [3]: ~ 5%

Example: Goal-oriented RE (“GORE”)
Papers published [1]: 966
[2] Mavin, et al. Does Goal-Oriented Requirements Engineering Achieve its Goal?, 2017
[3] Mendez et al. Naming the Pain in Requirements Engineering Initiative - www.re-survey.org
[1] Horkoff et al. Goal-Oriented Requirements Engineering: A Systematic Literature Map, 2016
Papers including a case study [1]: 131
Studies involving practitioners [2]: 20
Practitioners actually using GORE [3]: ~ 5%
For comparison:
Icelanders believing in elves [4]:
[4] https://www.nationalgeographic.com/travel/destinations/europe/iceland/believes-elves-exist-mythology/
54%

“[…] judging a theory by assessing the number,
faith, and vocal energy of its supporters […]
basic political credo of contemporary religious
maniacs”
— Imre Lakatos, 1970
Current state of evidence in SE is weak
* Addressing the situation in the quantum mechanics research community, an analogy

Current state of evidence in SE is weak
Source (levels of evidence): Wohlin. An Evidence Proﬁle for Software Engineering Research and Practice, 2013.
In favour /
corroboration
Against /
refutation
Strong evidence
Evidence
Circumstantial
evidence
Third-party claim
First or second party claim
Strong evidence
Circumstantial
evidence
Third-party claim
Evidence
First or second party claim
+
-
In most cases,  
we are here
Available studies often…
• … remain isolated
• … don’t report negative results
• … strengthen conﬁdence on  
own hopes (and don’t  
report anything around)
• … discuss little (if at all)  
relation to existing evidence

SE largely dominated by conventional wisdom
“Leprechauns”: Folklore turned into facts
• Emerge from times where claims by
authorities were treated as “facts”
• Reasons manifold:
– Lack of empirical awareness
– Neglecting particularities  
of practical contexts
– Neglecting relation to existing
evidence
– No proper citations (one side the
medal, over-conclusions, etc.)
– Lack of data
– …

Consequences
» Practical relevance and impact?
» Potential for transfer into practice and adoption?

The essence
Theory building and theory evaluation  
are crucial in SE research
» Reason about the discipline and (e.g. social)
phenomena involved
» Recognise and understand limits and effects of
artefacts (technologies, techniques, processes,
models, etc.) in their contexts

‣Theories in a Nutshell
… where we will brieﬂy talk about the general notion of theories
‣State of Evidence in Software Engineering
… where we will see why theory building is so important to our ﬁeld
‣Research Methods in Software Engineering
… where we will put research methods in a larger (philosophical)
picture
‣Qualitative “vs” quantitative research
… where we will discuss different research approaches

Recap: The Empirical Lifecycle
Source: Mendez and Passoth. Empirical Software  
Engineering: from Discipline to Interdiscipline, 2018.

(Empirical) methods
•Each method…
– …has a speciﬁc purpose
– …relies on a speciﬁc data type
•Purposes
– Exploratory
– Descriptive
– Explanatory
– Improving
•Data Types
– Qualitative
– Quantitative

(Empirical) methods
•Each method…
– …has a speciﬁc purpose
– …relies on a speciﬁc data type
•Purposes
– Exploratory
– Descriptive
– Explanatory
– Improving
•Data Types
– Qualitative
– Quantitative
“Grounded Theory”
Qualitative Data
Descriptive,
Exploratory, or
Explanatory
Example!

(Empirical) methods - where do they belong?
Ethnographic studies,
Folklore gathering
Case studies / 
Field studies
(Conﬁrmatory)
Formal analysis /
logical reasoning
Survey research
Case studies / 
Field studies
(Exploratory)

Which research methods to use  
in which situations?

Empirical process: an abstract view
Planning and Deﬁnition
Method and Strategy
Selection
Design and (Method)
Execution
Conclusion Drawing
Packaging and
Reporting
•Identify and outline problem (area)
•Determine research objectives and questions
•Select type of study and method(s)
•Identify necessary environment  
(including units of analysis)
•Design and validate study protocol (and validity
procedures)
•Collect & analyse data following detailed processes
•Interpret results
•Reﬂect upon potential threats to validity
•Package and (ideally) disclose data
•Report on results in tune with audience

Empirical process: an abstract view
Planning and Deﬁnition
Method and Strategy
Selection
Design and (Method)
Execution
Conclusion Drawing
Packaging and
Reporting
Scope of detailed empirical methods
The selection of proper research  
methods is a recognised problem  
in the community

There is no such thing as a  
universal way of scientiﬁc practice!

Method selection depends on many  
non-trivial questions
• What is the purpose of the study?
– Exploratory? Descriptive? Explanatory? Improving?
• What is the nature of the study?
– Inductive? Deductive?
• What is the relation to existing evidence?
– Building a new theory? “Testing” existing theory?
• What is the nature of the questions we ask?
– What-questions? Why-questions?
• What is the nature of the environment?
– Controlled environments? Realistic environments?
• What is the necessary sample?
– Population source?
– Units of analysis?
Criteria for  
selecting methods
Criteria for  
environment selection 
(and sampling)
Non-exclusive andnon-sequential

Not trivial, but possible: By using checklists
More advanced
• Chapter 16 + Appendix
• http://bit.ly/checklists-design_science
Good starting point

How to achieve scientiﬁc progress?
In step-wise iterations, with multiple methods  
(aka “research programme”)

Progress via multi-study approaches
Large-scale evaluation
5
e.g. Field Study or longitudinal study
1
2
3 4 5
Validation of new technology
in artiﬁcial setting
3
e.g. Controlled Experiment
Evaluation of new technology
in realistic setting
4
e.g. Case Study
Replication
Replication
Proposal new / adaptation
existing technology
2
Problem analysis
1
e.g. Systematic Mapping Study 
or Survey
e.g. RE Improvement  
Approach

Postulate I
Every study has a speciﬁc scope of validity only
Source: Sjøberg, D., Dybå,T.,Anda, B., Hannay, J. Building Theories in Software Engineering, 2010.

Postulate II
Scope of validity ≠ Degree of reality
Simulation
Field Study  
Research
Case Study  
Research
Survey
Research
Artiﬁcial environment
Realistic environment
Scope of validity*Controlled (lab)
Experiment
Replications
Replications
* Extremely simpliﬁed view to orient discussions

Postulate III
Case studies and experiments complement
each other in scaling up to practice
Based on: Wieringa R. Empirical Research Methods for TechnologyValidation: Scaling Up to Practice, 2013.
Lab credibility
Street credibility
Simple model
Realistic case
Small sample Large sample
Focus of  
case
studies
Focus of ﬁeld studies  
and replications
Scaling up to practice
Focus of  
(lab) experiments
Similarity to  
population units
Sample size

What is the difference between qualitative
and quantitative research?
What do you think?

Warning: EmSE emerges from natural science,
thus, qualitative methods are often confronted
with prejudice “I prefer working with real data  
[not with qualitative data]”
— Anonymous ISERN member
“In contrast [to previous qualitative studies], this study attempts
to obtain more scientiﬁc evidence in the form of objective,
quantitative data.”
— Anonymous author
“With all due respect, DO NOT make such ridiculous claims that only quantitative
studies are "scientiﬁc" and “objective" There is NO more objectivity in numbers
than there is in qualitative data.
”
— Anonymous reviewer
… response

The essence
Qualitative studies focus primarily on the kind of evidence
that will enable you to understand the meaning [and purpose,
reasoning, etc] of what is going on.
Quantitative studies focus primarily on the kind of evidence
that will enable you to understand what is going on.
Qualitative and quantitative methods have complementary
purposes, strengths, and limitations in theory building.

Two complementary approaches (+1)
•Quantitative research
– Describing events and ﬁnding causes to predict similar events in the future
– (Typically) focus on what, how much, or how many
– (Typically) in numerical forms
– (Typically) descriptive purpose
•Qualitative research
– Understanding meaning [and purpose, reasoning, etc.] of a phenomenon for
those involved
– (Typically) focus on why/meaning, and how people interpret their experiences
– (Typically) in variety of non-numerical forms, like texts, diagrams, etc.
– (Typically) exploratory or explanatory purpose
•Mix-method research
Adopted from: Da Silva. Tutorial given at the Ibero-American Conference on Software Engineering, 2018 (Bogota, Colombia)

Quantitative data Qualitative data
“Why?”-Questions“What?”-Questions
Descriptive & predictive purposes Explanatory & exploratory purposes

Quantitative data Qualitative data
Case study research
Survey research
(Quasi-) Controlled  
experiments
Action  
research
Ethnographic
studies
“Why?”-Questions“What?”-Questions
Descriptive & predictive purposes Explanatory & exploratory purposes

Comparison: qualitative “vs” quantitative
Qualitative Research Quantitative Research
Goals • Understanding, reasoning,
explanations, descriptions, meaning
(to subjects), discovery
• Hypothesis generation (typically)
• Description, control, prediction
• Hypothesis testing (typically)
Design
characteristics
• Flexible, evolving, emergent
• Inductive, constant comparative
• Predetermined, structured / ﬁxed
• Deductive, statistical
Samples • Small, non-random (sometimes even
opportunistic), purposeful,
theoretical
• Large(r), random, representative
Data collection • Researcher often primary instrument
• Interviews, observations, document
analysis, …
• Inanimate instruments (tests,
surveys, questionnaires, etc.)
Findings • Comprehensive, holistic, rich
descriptions
• Precise and statistical
* For all, you can add a “in tendency”

Researchers’ necessary skills
•Being a careful observer with
high tolerance to ambiguity and
existing belief systems
•Ability to ask “proper”, context-
speciﬁc questions without
actively distorting results
•Thinking inductively
•Patience (and charm)
•Comfort with observing  
(and writing :-)

Challenges in conducting qualitative studies
• Cases/subjects need to offer rich insights
(challenging in industry contexts) —>
Case selection often opportunistic
• Data collection demands skills beyond
statistics (e.g. social skills)
• Data analysis is often a (pragmatic and)
creative task.
• Data disclosure particularly important
(and challenging for ethical and legal
reasons)
• Communication and publication often
rendered difﬁcult by prejudice
Source: Mendez, Wagner. Case Studies in Industry: What we have Learnt, 2016

Further reading: Selected Papers
Good starting point Terminological demarcation
and key characteristics  
of methods
Epistemological setting

‣ Theories are the backbone of our
discipline allowing us to move forward
from paradigmatic stage of an
engineering discipline to a scientiﬁc one.
‣ Every research method has its place in a
larger picture.
Thank you!
‣ Qualitative and quantitative research has
complementary purposes, strengths, and
limitations in building and evaluating
theories.

Building and Evaluating Theories   in Software Engineering

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building and Evaluating Theories   in Software Engineering

Similar to Building and Evaluating Theories   in Software Engineering (20)

More from Daniel Mendez

More from Daniel Mendez (10)

Recently uploaded

Recently uploaded (20)