Case studies in industry - fundamentals and lessons learnt

Fundamentals and
Technische Universität München
Case Studies in Industry
Daniel Méndez
!
!
@mendezfe
Technische Universität München
Germany
!
IASESE 2014
Torino, Italy
Based on material from a joint work with:
Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering)
Stefan Wagner (University of Stuttgart)
!
Feedback from
Antonio Vetrò (Technische Universität München)

What is a case study?
Empirical inquiry that investigates a contemporary phenomenon within its real-life context,
especially when the boundaries between phenomenon and context are not clearly evident.
[Yin 2002]

In quest for a case study definition
We opt for realistic environments
Theoretical space
(methods, models)
Theory
Environment made
irrelevant
Artificial (controlled)
environment
Realism
Reality

We rely on a mix of qualitative & quantitative data
Case studies
Surveys and interviews
Controlled experiments
Quasi experiments
Action
research
Grounded
Theory
Quantitative Qualitative

We rely on a mix of empirical methods
Data collection procedure
Case studies
Action
research
Data analysis procedure
Grounded
Theory
Surveys and
interviews
* Example
* *

Summary of some important characteristics
!
!
Most* case studies:
• are conducted in socio-economic contexts
• are of confirmatory nature (“Let’s see if my method is better than yours”)
• rely on qualitative data (e.g. as part of interview transcripts)
• rely on a mix of various research methods, often (technical) action research
Case study research
Setting
Purpose
Data type
Mixed w/
•Realistic environments
•Realistic models
•Confirmatory
•Exploratory
•Qualitative
•Quantitative
* Unsupported claim, don’t sue me.
•Action research
•Interview research
•…

Agenda
1. Case study research: Why and how?
2. Examples
3. Challenges in case study research
4. Time for questions

Agenda
1.Case study research: Why and how?
2. Examples

Why case studies?
Learn from real life environments
Disclaimer
Cases studies do not imply more
“validity” or “trustwor thiness” than
experiments - they complement each
other!
Learn from realistic (socio-economic) environments with realistic conditions:
• subjects with: opinions, experiences, expectations, fears, beliefs, …
• cases with: realistic models, realistic complexity, realistic decisions, …
… at the cost of control (and internal validity)
!
» Ideally: Explore the unpredictable (e.g. when testing a SE method)

Why case studies?
Case studies support scaling up to practice
Lab credibility
Street credibility
Focus of case studies
Realistic model
Simple model
Small sample Large sample
Based on: Wieringa
Empirical Research Methods for Technology Validation: Scaling Up to Practice
(Focus of field studies
and replications)
Scaling up to practice
Focus of
(lab) experiments

What is the lifecycle of a case study?
Case study
design
Preparation for
data collection
Data collection Data analysis Reporting
• Research objectives
• Research goals
• …
• Schedules
• Protocols
• Document analyses
• Interviews, Surveys
• Action research
• …
• Descriptive statistics
• Grounded theory
• …
• Packaging
• (Publishing)
Further reading

Agenda
2.Examples

Model-based process enactment
Develop and evaluate an approach to dynamically test the conformance of
software project (instances) and software process models
Problem analysis Evaluation
Development
(TAR*)
Case study
research
Interviews
Document
analysis
* Technical Action Research

Results
Development
(TAR)
Case study
research
Interviews
Document
analysis
• Moving targets detected when designing the instruments for the interviews
• New goals had no scientific relevance (very low theoretical and practical impact)

Model-based requirements engineering
Develop and evaluate a model-based requirements engineering approach for embedded reactive
systems in the automotive domain
Development
(TAR)
Case study
research
Interviews
Document
analysis

Results
Development
(TAR)
Case study
research
Interviews
Document
analysis
No access to software project environments (“The ivory tower”)

Model-based requirements engineering (2nd try)
Evaluate a model-based requirements engineering approach for business information systems
in comparison to previously used approach
Case study
Assessment
(Interview)
TAR
workshops

Case study
Assessment
(Interview)
TAR
workshops
Results
The new model-based approach supported:
• more flexibility in process
• higher productivity
• higher syntactic quality in artefacts
!
Conclusion:
The new model-based approach better supports
a flexible process with high quality artefacts.
See also: Mendez Fernandez et al.
A Case Study on the Application of an Artefact-based Requirements Engineering Approach
!

Conclusion:
The new model-based approach better supports
a flexible process with syntactically consistent results.
What problems do you see in this case study (and conclusions)?
• Action research workshops inherently threaten the internal validity
• High degree of subjectivity
• Results sensitive to their context (generalisation?)
• …
!
» Each “successful” case study still offers plethora of problems and challenges!

Agenda
2. Examples
3.Challenges in case study research

Industrial research cooperations with
• up to 3 years duration
• 3-30 participants
• multiple case studies
We distinguish three classes of challenges:
1. Approaching a case study
2. Conducting a cases study
3. Transversal challenges
(…)

Challenges in approaching a case study

#1: Finding contacts
» Integrate studies into ongoing / planned research projects
» Actively approach
– alumni of your university
– companies in your area
– practitioners at (local) events
Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1
!

#2: Convincing contacts to participate
» Find appropriate terminology (technology transfer)
» Find appropriate incentives for project partners
» Respond to their local needs
» Provide early feedback / results
» In funded projects, probability is higher that practitioners collaborate

#3: Opportunistic case and subject selection
» Welcome to case study research!
Image source: xkcd
!

#4: Planning a case study
You need to:
• schedule meetings / workshops
• allocate resources
• (formally) define deliverables
• clarify responsibilities
• clarify NDA-related issues
• …
(the list can get long)
!
» Plan a case study like you’d plan a software development project:
– Time plan and resource allocation
– Quality assurance and acceptance of protocols and reports
– …
Image source: https://c1.staticflickr.com/1/186/467087455_e90b880c92.jpg
!

#5: Uncertainty
• What will the stakeholder characteristics be?
• Availability
• Skills
• Motivation to participate
• Commitment to goals
• Representativeness
• …
• What will the data look like?
•Quality of the data
•Quantity of the data
» Take early samples and test them!
» Test instruments and data quality via pilots
» Get to know your subjects early
» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”

Challenges in conducting a case study

Context variables (and phenomena)
• hard to determine (or unavailable)
• hard to measure
» Context often hard to describe
(hence, hard to reproduce by others)
Variables described
in report
#1: Properly characterising the context
» Orient yourself on
Variables you
should measure
Variables you
can measure
» other (similar) studies or
» on classification schemes from the area of software process models (“Tailoring”)
» Focus on whole context, not only on cases for which you want to draw conclusions
» Ask yourself: “What information would I need to understand and replicate the study?”

#2: Communication
Possible problems:
• No clear communication of case study in organisation
• Case study delegated to s.o. not interested
• No clear communication lines and reporting lines
» Define clear communication/reporting/escalation lines at the beginning. Take no shortcut!
» When communicating with subjects:
» Communicate purpose and goals of case study
» Find a supporter at project level (with own interests)
» Don’t call them subjects ;-)
Image source: https://c1.staticflickr.com/1/32/62630024_10a8fa36ce_z.jpg?zz=1
!

#3: Defining the proper instruments
Possible problems:
• Wrong assumptions about population
• (Research) questions not
thought through
» Pre-tests your design, e.g.
» Pilot your surveys
» Check if planned data analysis techniques apply
» A way to calibrate the instruments is to conduct a small experiment with students first
Image source: How to lie with statistics
!

#4: The data collection might take long…
» Be patient
» Never push!
Image source: https://c1.staticflickr.com/9/8185/8375141385_df9bd4f283_z.jpg
!

#5: Working with sensitive data
» Clarify the rules:
» what you need for the case study
» what can be stored and reported (internally and externally)
» Plan review and acceptance cycles for reports and publications
» The “easier” way (sometimes): Joint publication

#6: (Social) skills
Possible distortions in data collection:
• biased researchers
(trying to sell own PhD topic)
• weak moderation skills
• hidden agendas of respondents
(missing trust!)
• (…)
» Don’t be biased!
» Involve other researchers in:
» Design of instruments
» Data collection (e.g. interviews)
» Continuously practice your
» Moderation skills
» Rhetoric and listening skills
» Build trust by showing loyalty and respect

#7: Analysing qualitative data
Qualitative data
• is inherently subjective
• is difficult to interpret
(we all have different mental models!)
» When analysing qualitative data
» Work in teams (researcher triangulation)
» When making interpretations, document the rationale
» Validate the results with the subjects (if possible)
» Be careful when drawing conclusions
Image source: How to lie with statistics
!

#8: Moving targets
• Moving targets are something natural
• The difficult part:
• Detecting moving targets
• Reacting to them
» Continuously provide and get feedback
» When planning your study, agree on
» key goals and questions
» open space left for new research questions (flexible part)
» Goals should always meet the interest of both industry and academia

#9: Tool support
Unfortunately,
still the best tools we have…

Transversal challenges
#1: The ivory tower
Possible problem:
Support by management…only by management.
» No backup in projects (organisational culture)
» Make clear study plan including needed
subjects & cases (and formally agree on it)
» Talk early to subjects
» Find active supporter (not sponsor) with own
interests in project (= their own incentives)

#2: Sometimes, size seems to matter…
» „[Even if] one swallow does not make a summer, one swallow does...
» prove the existence of swallows
» provide a great deal of reliable information about swallow anatomy“*
» Do not be that kind of reviewer that asks for both realistic models and large sample sizes
* Y. Anzai and H.A. Simon. The theory of learning by doing (1979) / H. Sharp. Keynote (EASE 2012)

#3: Drawing proper conclusions
» Knowledge is more than statistical significance
» When analysing the results, be accurate and critical!
» Stay on the ground
» You may interpret, as long as the interpretation is clearly stated as such
» Rely on other case studies that might explain your observations
Image source: xkcd
!

#4: The quest for universal truth (“generalisation”)
1. Software engineering is about human beings (Subjectivity)
2. Reality cannot be captured in complete system of variables
!
» “Truth” will always be something relative to its context
» Aim for trustworthiness and transparency rather than for int./…/ext. validity
» Support scaling up to practice
» Appreciate the value of replication studies
» Align your study with similar ones
» Make clear relation to existing evidence
» Disclose your data (e.g. via promisedata)
See also: Mendez et al.
In Quest for Requirements Engineering Oracles: Dependent Measurements and Variables for (good) RE

#5: Openness and transparency
Case study data should be disclosed
• reliability and trustworthiness?
• reproducibility and replicability?
» The success of empirical SE relies on a
collaborating community
» If possible, make your (anonymised) data
accessible to others

#6: Properly reporting the results
Case study results need to be disseminated
• to practice
• to academia
» How to properly report?
» Report results according to target group:
» Small presentations to management w/ focus on results
» Peer-reviewed publication w/ (selected) research scope
» (Technical) Reports with full data (and analysis)
» …
» Rely on established guidelines for reporting on case studies
!

My personal manifesto for case study research
6. Be unbiased
7. Be accurate and critical
8. Be honest
9. Be patient
10. Be flexible (and pragmatic!)
1. Carefully plan your study like a project
2. Make clear goals, purpose, context, threats
3. Train your empirical and social skills
4. Disclose your data (reliability & replicability)
5. Collaborate

Agenda
2. Examples
4.Time for questions

!
!
!
Thank you!
Daniel Méndez
Daniel.Mendez@tum.de
@mendezfe
Further reading
• Slides will be made available
(and probably tweeted)
• Ask me if you need material
(studies, templates, …)

Case studies in industry - fundamentals and lessons learnt

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Case studies in industry - fundamentals and lessons learnt

Similar to Case studies in industry - fundamentals and lessons learnt (20)

More from Daniel Mendez

More from Daniel Mendez (15)

Recently uploaded

Recently uploaded (20)

Case studies in industry - fundamentals and lessons learnt