Validity and Reliability of Qualitative Assessments based on self-reported statements

Validity and Reliability of Qualitative Assessments Based on Self-reported Statements
Validity and Reliability of Qualitative Assessments
Based on Self-reported Statements
Alexander Haarmann
QMM2015
2 October 2015
Alexander Haarmann 2 October 2015 1 / 21

Background
PhD thesis on emergence of formal collective patient and public
involvement (PPI) in (health-)care

Background
PhD thesis on emergence of formal collective patient and public
involvement (PPI) in (health-)care
the Netherlands (all in-patient care) & England (Foundation Trust
Hospitals) have established almost identical means of collective PPI
user councils as additional tier in governance
advisory character, few veto rights
virtually identical rights

Theory & Reality

Theory & Reality
Healthcare Types
Typology according to.. .
Neubauer and
Birkner (1984)
Wendt and Rothgang (2007) Moran (2000) Esping-
Andersen (1990)
• Pay on your
own
(Predominantly Character-
ised by) Private Healthcare
- Supply
Liberal
• Charity
Provision (Predominantly) State-run/
National Healthcare System
Entrenched/ Command
& control
Social-
democratic
Insurance (Predominant) Logic of So-
cial Insurance
Corporatist Conservative

Theory & Reality
Healthcare Types
Typology according to.. .
Neubauer and
Birkner (1984)
Wendt and Rothgang (2007) Moran (2000) Esping-
Andersen (1990)
• Pay on your
own
(Predominantly Character-
ised by) Private Healthcare
- Supply
Liberal
• Charity
Provision (Predominantly) State-run/
National Healthcare System
Entrenched/ Command
& control
Social-
democratic
Insurance (Predominant) Logic of So-
cial Insurance
Corporatist Conservative
postulated relation between funding, provision, & governance
nevertheless same type of PPI in England & the Netherlands

Resulting Questions
What factors can be identiﬁed to have contributed to or hampered
the implementation of collective patient involvement in hospitals?

Resulting Questions
What factors can be identiﬁed to have contributed to or hampered
the implementation of collective patient involvement in hospitals?
Have the main goals pursued with the respective legislation
been achieved according to key actors in the ﬁeld?

Methodology & Sources
explorative, qualitative, comparative study with the main cases England
& the Netherlands, which are compared to Sweden & Germany

Sources
all kinds of literature, including grey literature
green & white papers
parliamentary minutes
legal texts

Sources
legal texts
expert interviews (nine to 15 interviews per country; on average
more than 50 minutes recorded length)
politicians of ruling & opposition parties at time of enactment
(health-)care managers (acute, mental, ambulance; long-term)
members of user councils
patient organisations
researchers

Sources
legal texts
expert interviews (nine to 15 interviews per country; on average
more than 50 minutes recorded length)
politicians of ruling & opposition parties at time of enactment
(health-)care managers (acute, mental, ambulance; long-term)
members of user councils
patient organisations
researchers
participant observation

Question of the Presentation
How valid & reliable is the collected data?
How suitable is it for evaluating changes, policies,
implementations?

Data to Compare With
internal
expert interviews vs. participant observation
contrasting statements of actors/ professional groups
external
NL: a number of reports, surveys, & broader range of scientiﬁc
contributions (e. g. Savornin Lohman 2000; Hoogerwerf, Nievers
and Scholten 2004; LSR 2007; Lammerts et al. 2008b; Lammerts
et al. 2008d; Lammerts et al. 2008c; Lammerts et al. 2008a;
Oudenampsen et al. 2008; Trappenburg 2008)
England: few anecdotal reports, one survey (Ham and Hunt 2008;
Ipsos MORI 2008; Bojke and Goddard 2010; House of Commons
Health Committee 2008a; House of Commons Health Committee
2008b)

Results (NL)
Sectoral Differences
works best in institutions of long-term care
sticking point hospitals
different traditions

Results (NL)
“I think, that [...] they [the client councils] do not have a strong position
in hospitals [...]. [S]uch a council of clients is for most people a reality
that is far away. [...] And I think, to be honest, if I were responsible in
the Hague as minister nowadays I would rethink about a role of the
client councils. The function is taken over by others. [...] That is
something different for an old-age home or a nursing home, or for a
mental care institution, that is substantially different. Even for a
primary care institution in a village or a city with a dentist and a GP, and
social work etc.” (interview 11; author’s translation)

Results (NL)
Frequency of Informing Other Patients/ Residents in Percent
Mental Care Rehab Facility Home Care Hospitals
once a year 12 n. a. 39 41
once half a year 9 n. a. 13 13
once a quarter 30 25 13 6
once a month 33 n. a. 3 2
other 16 n. a. 32 37
n 43 11 31 46
Sources: Lammerts et al. 2008a, p. 28; Lammerts et al. 2008b, pp. 28–29; Lammerts et al. 2008c,
pp. 26–27; Lammerts et al. 2008d, p. 28

Results (NL)
Involving Grassroots in Advice in Percent
Mental Care Rehab Facility Home Care Hospitals
Visiting clients 70 36 30 4
Surveys 56 54 60 58
Contact person 48 36 33 31
Meetings 46 27 18 31
’Ideas Boxes’ 32 63 9 –
Website 23 54 39 40
Telephone 23 27 21 11
Other 32 54 42 51
n 43 11 30 46
Sources: Lammerts et al. 2008a, p. 30; Lammerts et al. 2008b, p. 29; Lammerts et al. 2008c, p. 27;
Lammerts et al. 2008d, p. 29

Results (NL)
different topics
rehab & hospitals: budget & organisational changes
home care: quality of care & complaint procedures
psychiatric care: closure/ extension of wards, quality of services,
food & budget
Lammerts et al. 2008b; Lammerts et al. 2008d; Lammerts et al. 2008c;
Lammerts et al. 2008a

Results II (NL)
Learning Curve
change over time, learning curve of 10–15 years

Results II (NL)
Learning Curve
change over time, learning curve of 10–15 years
process from “What are our rights?” to “What issues would we like to
raise?”
ﬁnd a common language
different lifeworlds
“Look, such a client council, they do not want to talk a full hour about the
temperature of the tea. However, if it feels it is not heard – that keeps
coming back. And the member of the executive board keeps thinking:
‘Why do I sit here?’ and tries to suck the client council with him into his
world of the system, of control, of long-term perspective, whereas the
client council has the perspective on the short-term.” (interview 1;
author’s translation)

Results III (NL)
Different Perspectives
heterogeneous statements of different actors/ groups

Results III (NL)
virtually everyone agrees that hospital sector is a speciﬁc case

Results III (NL)
scientists & politicians rather positive
“Thus, they have actually very much inﬂuence on the policies,
in case they claim it and do not let the management do with
them.” (interview 10; author’s translation)

Results III (NL)
users’ umbrella organisations: 1
3 positive, a few bad examples, &
many in between extremes
“The problem is always that the client council needs to collaborate
with the executive board. However, if you come to a controversy with
each other it does not get better for anyone! [...] It is difficult to [say]:
We demand this and we demand that, since then such an executive
board says: There is no money for that! You may demand whatever you
want! [.. .] That will not happen!” (interview 7; author’s translation)
“How do you deal with the flow of information from the board of
directors? They dump it, let the council of clients keep busy. [...] You
can flood the councils with information. And boards of directors have
a knack for it.” (interview 3; author’s translation)

Results III (NL)
users’ umbrella organisations: 1
3 positive, a few bad examples, &
many in between extremes
members of user councils & management agree: cooperation –
and, thereby, user involvement – works fairly well
perceived inﬂuence: in general between 4 & 5 (on 5 point scale)
not all boards give requested advice, only few unrequested
Sources: Lammerts 2008; Lammerts 2008a; Lammerts 2008b; Lammerts 2008c

Results (England)
England
similar picture as in the Netherlands, similar problems
PPI liaison managers & politicians more critical than in NL
difficulties to recruit a sufficient number of members and
governors, in particular in mental care
governors’ roles are ill-defined
results are mirrored in survey and reports

Summary of Comparison
Comparison with survey results, reports show strengths and weaknesses
=
objective information about size & composition of council, frequency of meetings
etc.
differences between actors
developmental aspects
the overall picture

=
etc.
the overall picture
–
representativity
prone to sampling error

=
etc.
the overall picture
–
representativity
+
information & more subtle differences going beyond standardised items; e. g.:
learning curve of 10–15 years to “understand” each other
following the rules can mean infringing the original intention

=
etc.
the overall picture
–
representativity
+
information & more subtle differences going beyond standardised items; e. g.:
learning curve of 10–15 years to “understand” each other
following the rules can mean infringing the original intention
⇒ Comparison seems proof for high reliability & validity of
expert interviews

Comparison with Internal Data
Comparison of results from expert interviews with content of
participant observation during internal council meeting
Expert interview Participant observation
•good director & good
relation to management
•lack of support by
administration
•management is supportive •relevant papers are
repeatedly distributed too
late
•council is taken seriously
& can inﬂuence quite a lot
•council is not taken
seriously & needs to step
up

Potential Reasons for Discrepancies
potential reasons might consist of a range of social-psychological
mechanisms such as:

potential reasons might consist of a range of social-psychological
mechanisms such as:
social desirability
self-praise
managers proﬁt from portraying themselves as listening &
understanding
users proﬁt from portraying themselves as active & successful
presenting PPI as a success story to the foreign researcher
maybe actual improvement over situation some 25 years ago
...

Further Steps!?
age-old question about validity

Further Steps!?
the problem is not to be found in a speciﬁc method, but in a
speciﬁc combination of topic and/ or actor
⇒ mixed methods design does not increase validity per se

Further Steps!?
the problem is not to be found in a specific method, but in a
specific combination of topic and/ or actor
⇒ mixed methods design does not increase validity per se
better understanding of underlying processes is needed
long-term perspective necessary? one-off shot of reality sufficient?
potential ways to handle problem:
posing much more specific questions
mixed methods designs that raise the validity of the data
different cultures of access in various countries

Thank
you
for
your
attention!
Contact details:
Alexander Haarmann
University of Bremen, BIGSSS
ahaarmann@bigsss.uni-
bremen.de
alexander.haarmann@web.de

Table of Contents
1 Starting Point
Background
Contradiction between Theory & Reality
Resulting Questions
Methodology
2 Question of the Presentation
Data to Compare With
3 Results
Results (NL)
Summary of Results
Comparison with Internal Data
4 Conclusions
Further Steps?
References

Bibliography I
Bojke, Chris and Maria Goddard (Sept. 2010). Foundation Trusts: A Retrospective Review. Centre for Health Economics,
University of York. York.
Esping-Andersen, Gøsta (1990). The Three Worlds of Welfare Capitalism. Princton: University Press.
Ham, Chris and Peter Hunt (July 2008).
Members Governance in NHS Foundation Trusts: A Review for the Department of Health.
Hoogerwerf, Remco, Eline Nievers and Christel Scholten (July 2004).
De invloed van cliëntenraden – Het verzwaard adviesrecht in de dagelijkse praktijk vanuit cliëntenraadperspectief. research
voor beleid bv. Leiden.
House of Commons Health Committee, ed. (17th Oct. 2008a). Foundation trusts and Monitor. Sixth Report of Session 2007–08
Volume I: Report, together with formal minutes.
— ed. (17th Oct. 2008b). Foundation trusts and Monitor. Sixth Report of Session 2007–08 Volume II: Oral and written evidence.
Ipsos MORI (Jan. 2008).
Survey of Foundation Trust Governors – Research Study Conducted for Monitor – Independent Regulator of NHS Foundation Trusts.
London.
Lammerts, R., L. M. Verwijs, J. Homberg, K. D. Hekkert, S. J. W. de Kroon and H. C. M. Kamphuis (2008a).
Patiënten-, gehandicapten- en ouderenorganisaties in Nederland – Cliëntenraden in beeld – Cliëntenraden van GGz-instellingen in 20
Utrecht: LSR.
— (2008b).
Patiënten-, gehandicapten- en ouderenorganisaties in Nederland – Cliëntenraden in beeld – Cliëntenraden van revalidatiecentra in 20
Utrecht: LSR.

Bibliography II
Lammerts, R., L. M. Verwijs, J. Homberg, K. D. Hekkert, S. J. W. de Kroon and H. C. M. Kamphuis (2008c).
Patiënten-, gehandicapten- en ouderenorganisaties in Nederland – Cliëntenraden in beeld – Cliëntenraden van thuiszorgorganisaties i
Utrecht: LSR.
— (2008d).
Patiënten-, gehandicapten- en ouderenorganisaties in Nederland – Cliëntenraden in beeld – Cliëntenraden van ziekenhuizen in 2008.
Utrecht: LSR.
LSR (Mar. 2007). een onderzoek naar de praktijk van de wmcz in ziekenhuizen en revalidatieinstellingen. Utrecht: LSR.
Moran, Michael (June 2000). “Understanding the welfare state: the case of health care”. In:
British Journal of Politics and International Relations 2.2, pp. 135–160.
Neubauer, Günter and Barbara Birkner (1984). “Alternativen der Organisation und Steuerung des Gesundheitswesens”. In:
Alternativen der Steuerung des Gesundheitswesens im Rahmen einer Sozialen Marktwirtschaft. Ed. by Günter Neubauer.
Gerlingen: Robert Bosch Stiftung, pp. 13–70.
Oudenampsen, Dick, Helen Kamphuis, Rob Lammerts, Jeroen Homberg and Esmy Kromontono (Dec. 2008).
Patiënten- en Consumenbeweging in Beeld. Utrecht: Nederlandse Patiënten Consumenten Federatie (NPCF).
Savornin Lohman, J. de (2000). “Evaluatie – Wet medezeggenschap cliënten zorgsector”. In:
Evaluatie Wet medezeggenschap cliënten zorginstellingen. Ed. by J. de Savornin Lohman. Utrecht: Verwey-Jonker Instituut.
Trappenburg, Margo (2008). Genoeg is genoeg. Amsterdam: Amsterdam University Press, p. 272.
Wendt, Claus and Heinz Rothgang (2007).
Gesundheitssystemtypen im Vergleich – Konzeptionelle Überlegungen zur vergleichenden Analyse von Gesundheitssystemen.
61. Universität Bremen, Sonderforschungsbereich 597 (Staatlichkeit im Wandel).

Validity and Reliability of Qualitative Assessments based on self-reported statements

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (8)

Similar to Validity and Reliability of Qualitative Assessments based on self-reported statements

Similar to Validity and Reliability of Qualitative Assessments based on self-reported statements (20)

More from Gemma Derrick

More from Gemma Derrick (13)

Recently uploaded

Recently uploaded (20)

Validity and Reliability of Qualitative Assessments based on self-reported statements