Presentation given by Tereza Stockelova and Sarah de Rijcke at the Workshop exploring Qualitative and Mixed Methods in Research Evaluation and Policy 2015 (QMM2015)

1. Tereza Stöckelová Sarah de Rijcke
Institute of Sociology, Czech Academy of Sciences CWTS, Leiden university
Rethinking the ‘international’ in the
governance of science
01 - 02 October 2015 Brunel University London 1

2. Outline
• The normative of the ‘international’
• Ethnography and evaluation studies
• The ‘international’ in CR and NL research policies and
assessment systems
• Normative hierarchies
• Next steps: triangulation
– Co-word analysis
– Interviews
• Rethinking the international
01 - 02 October 2015 Brunel University London 2

3. The normative of the ‘international‘
• ‘International‘ as the means and end of research policies
• Our joint project:
How does the normative framework play out in 2 smaller, non-
Anglophone countries with different (science) histories?
• Exploratory phase
– Document and discourse analysis
– Analysis of normative hierarchies
– The comparative moment, focus on similarities
• Next steps
– co-word analysis, interviews, ethnographic interventions
01 - 02 October 2015 Brunel University London 3

4. Ethnography and evaluation studies
• Key method for in-depth study of heterogeneous research
governance and assessment practices
– including the 'social life‘ and agency of policy documents, indicators,
protocols, categories, and classifications
• Method aims at understanding the logics and multiple possibilities
of practices under study
– Enables study of practices including the tacit, unofficial and messy; what is
not, or cannot be, accounted for verbally
– Focuses on meanings of actions and explanations, rather than quantification
– Theoretical, not statistical representativeness
– Highly reflexive approach – performativity of methods – never innocent
01 - 02 October 2015 Brunel University London 4

5. Czech Republic
01 - 02 October 2015 Brunel University London 5

6. THE INTERNATIONAL
• National research evaluation schemes developed and tweaked in the CR
since 2001; research assessment exercise takes place annually
• Since 2004 with consequences for institutional funding
• The international as a key reference
‘‘In funding basic research [we have to] take the citation count as a criterion
of success, mainly on the international scale. (…) Outcomes of basic research
which do not permeate into the ‘global market’ of science are in a similar
position as innovations that have already been invented and applied by
someone else.’’
Prime Minister of the Czech Republic on his government’s science policy and funding
priorities (2007)
01 - 02 October 2015 Brunel University London 6

7. THE INTERNATIONAL
01 - 02 October 2015 Brunel University London 7
CZ RESEARCH
CZ RESEARCH EVALUATIONIF, SCOPUS, INT. PATENTS

8. THE INTERNATIONAL
01 - 02 October 2015 Brunel University London 8
CZ RESEARCH
CZ RESEARCH EVALUATIONIF, SCOPUS, INT. PATENTS
CRITIQUE OF CZ RESEARCH EVALUATIONINT. POLICY STANDARDS NOT MET

9. THE INTERNATIONAL
01 - 02 October 2015 Brunel University London 9
CZ RESEARCH
CZ RESEARCH EVALUATIONIF, SCOPUS, INT. PATENTS
CRITIQUE OF CZ RESEARCH EVALUATIONINT. POLICY STANDARDS NOT MET
CRITIQUE OF THE RULE BY NUMBERS AND
‘AUDIT CULTURE‘
SAN FRANCISCO OR LEIDEN
DECLARATIONS, STS LITERATURE

10. THE INTERNATIONAL
01 - 02 October 2015 Brunel University London 10
CZ RESEARCH
CZ RESEARCH EVALUATIONIF, SCOPUS, INT. PATENTS
CRITIQUE OF CZ RESEARCH EVALUATIONINT. POLICY STANDARDS NOT MET
CRITIQUE OF RULE BY NUMBERS AND
AUDIT ‘CULTURE‘
SAN FRANCISCO OR LEIDEN
DECLARATIONS, STS LITERATURE
NEW RESEARCH EVALUATION SCHEME
(QUALI + QUANTI)
MODELLED ON RAE/REF, PANELS OF
INT. EXPERTS, CARRIED OUT BY
TECHNOPOLIS

11. 01 - 02 October 2015 Brunel University London 11
Criterion A: Institutional management and development potential
Starred Definition
quality level
4* Quality of the managements and the research
infrastructure is outstanding and beyond international
norms.
3* Quality of the managements and the research
infrastructure is state of the art from an international
perspective.
2* Quality of the managements and the research
infrastructure is adequate.
1* Quality of the managements and the research
infrastructure is not adequate at all point.

12. 01 - 02 October 2015 Brunel University London 12
Criterion B: Membership of the national and global research community
Starred quality Definition
level
4* The RU participates and is recognised in excellent
international networks involving the international
leaders in the field
3* Participates and is recognised in international and
national networks
2* Participates only or mostly in national networks
1* Participates only or mostly in regional or local networks

13. 01 - 02 October 2015 Brunel University London 13
Criterion C: Research excellence
Starred quality Definition
level
4* Quality that is world leading in terms of originality,
significance and rigour.
3* Quality that is internationally excellent in terms of
originality, significance and rigour but which nonetheless
falls short of the highest standards of excellence.
2* Quality that is recognised internationally in terms of
originality, significance and rigour.
1* Quality that is recognised nationally in terms of
originality, significance and rigour
Unclassified Quality that falls below the standard of nationally
recognised work

14. 01 - 02 October 2015 Brunel University London 14
Criterion E: Societal relevance
Starred Definition
quality level
4* In the context of its field, the work has outstanding
impacts in terms of reach and significance
3* In the context of its field, the work has very
considerable impacts in terms of reach and
significance
2* In the context of its field, the work has considerable
impacts in terms of reach and significance
1* In the context of its field, the work has modest
impacts in terms of reach and significance

15. AN INTERNATIONAL EVALUATION
OF LOCAL(IZED) SCIENCE?
01 - 02 October 2015 Brunel University London 15
CZ RESEARCH
CZ RESEARCH EVALUATIONIF, SCOPUS, INT. PATENTS
CRITIQUE OF CZ RESEARCH
EVALUATION
INT. POLICY STANDARDS NOT MET
CRITIQUE OF RULE BY NUMBERS AND
AUDIT „CULTURE“
– IN DEFENSE OF LOCAL(IZED) SCIENCE
SAN FRANCISCO OR LEIDEN
DECLARATIONS, STS LITERATURE
NEW RESEARCH EVALUATION SCHEME
(QUALI + QUANTI)
– INTERNATIONAL EXCELLENCE,
DOMESTIC SOCIAL RELEVANCE
MODELLED ON RAE, PANEL OF INT.
EXPERTS, CARRIED OUT BY
TECHNOPOLIS CORPORATION

16. The Netherlands
01 - 02 October 2015 Brunel University London 16

17. Standard Evaluation Protocol (SEP)
• 2015 version: 5th iteration
• “describes the methods used to assess research conducted at
Dutch universities and NWO and Academy institutes every six
years, as well as the aims of such assessments.”
• In 1993 VSNU made responsible, 4-year cycle
• 2000s evaluation fatigue: 6-yearly. After advice VSNU, KNAW,
NWO. New version of protocol.
• Responsibility delegated to institutional level
01 - 02 October 2015 Brunel University London 17

18. Standard Evaluation Protocol (SEP)
• Based on peer review, informed by bibliometrics
• Up until 2009 four assessment criteria:
– Quality
– Productivity
– Societal relevance
– Vitality & feasibility
• New 2015 protocol:
– Research quality
– Relevance to society
– Viability
01 - 02 October 2015 Brunel University London 18

19. Standard Evaluation Protocol
(2009-2015)
01 - 02 October 2015 Brunel University London 19

20. 01 - 02 October 2015 Brunel University London 20
Standard Evaluation Protocol (2009-2015)

21. Comparing SEP 2009 and 2015
criteria
• 2009:
– *5: world-leading
– *4: internationally competitive, nationally leading
– *3: internationally visible, competitive nationally
• 2015
– *1: research unit is one of few most influential in the world
– *2: research unit is very good, internationally recognized
01 - 02 October 2015 Brunel University London 21

22. Comparing SEP 2009 and 2015
societal relevance
• 2009:
• Societal relevance: quality, impact and valorization
• Socio-cultural and/or technical or economic quality, impact,
valorisation
• ‘society’ geographically and hierarchically unspecified
• 2015:
• “The research unit adds a “narrative” to the output table of the
self-assessment. The purpose of the narrative is to explain the
relevance of the research unit’s work to society.”
• Distinction between regional/national/international level in
these ‘relevance narratives’
01 - 02 October 2015 Brunel University London 22

23. Comparing SEP 2009 and 2015
assessment committee
• 2009:
– Independent, acquainted with field, able to position research
within (inter)national context, experienced evaluators,
impartial (p. 5-6)
• 2015:
– International has become a sine qua non
• “The committee should be impartial and international.” (p. 6)
• “The committee takes into account international trends and
developments in science and society as it forms its judgement.” (p.6)
01 - 02 October 2015 Brunel University London 23

24. Normative hierarchies
• The international is enacted in the two national assessment
systems through hierarchical modes of ordering
– Research ‘excellence’ increasingly defined and
recognized through the ‘international’
• In rating systems
• In ‘internationalisation’ of assessment committees
– International research excellence, domestic societal
relevance
01 - 02 October 2015 Brunel University London 24

25. Modes of international
Lin & Law (2013): Making things differently: on ‘modes of international’
– analytical mode (Western biomedicine)
– correlative mode (Chinese medicine)
“What does it mean to be international? We’ve seen that it takes hard work to make
things transportable and hold them steady. This is a tough way of doing international,
but it’s pretty successful even so (…). Along with imperialism, colonialism, and
European gun-boat diplomacy in various shape and forms. And massacres. And opium
wars. And Japanese imperialism. And economic domination. And technological
subordination. And colonisation by ‘Western’ ideas, such that a Taiwanese university
campus looks very like one in Michigan with the same kinds of departments and
concerns: journal publication, rankings, the Science Citation Index, and all the rest.
But the making of similarities like this has been going on since Vasco da Gama in the
Western mode of international. Yes, it comes in various guises, as modernisation, or
Westernisation, or globalisation, or development, or even civilisation. But it’s been
done on a massive scale since the sixteenth century. And it has always been analytical
and dominatory.“
01 - 02 October 2015 Brunel University London 25

26. Rethinking and re-enacting
‘international’
What might a correlative research assessment look like?
Or the assessment supporting the production of correlative knowledge and
practices?
• multiplicity of places/perspectives of recognition and use that are not
hierarchically ordered from „local“ and „regional“ to „national“ to
„international“ but they would add up and calibrate each other;
• being explicit about where, geo-politically, the international is situated in
every concrete case (which would dissolve the international as an abstract
no-place);
• experimentally introducing symmetrical configurations in the enactment
of international (e.g. a view from the margin representing an international
perspective in relation to the centre) – could be also tried out in the form
of an ethnographic experiment (Mann et al. 2011)
01 - 02 October 2015 Brunel University London 26

27. Next steps
• Multiplying methods
– co-word analysis (cf. Derrick, Meijer & v. Wijk, 2014)
– tracing key international actors (such as Technopolis)
– Interviews
– Ethnographic interventions
• Towards more articulate modes of the international
01 - 02 October 2015 Brunel University London 27

28. ##
01 - 02 October 2015 Brunel University London 28

29. 01 - 02 October 2015 Brunel University London 29