This research in progress paper describes the initial results of a long-term, large-scale analysis of the operationalization of evaluation of the societal impact of research. Results from the first stage of qualitative interviews are used to illustrate the strength of the methodological design of the study.

1. Intentions and
strategies for evaluating
the societal impact of
research: Insights from
REF2014 evaluators
Derrick, G.E.
Health Economics Research Group
(HERG)
Brunel University London

2. Introduction
• UK REF2014 – national evaluation process of universities to distribute over £1,952
billion of government funding for research.
• Criteria: 65% Outputs (Peer review of publications), 15% Environment (Esteem) and 20% Impact.
• Impact is defined as research that has had “…an effect on, change or benefit to the economy,
society, culture, public policy or services, health, the environment or quality of life, beyond
academia.” (HEFCE, 2011)
• Peer review only accepted as legitimate IF the results of outcomes are perceived to
have been
• Evaluation of criteria conducted by 4 Main Panels divided into “Units of assessment” or
disciplines.
• For “Impact” evaluate 4 page case studies “in session” – ex-post impact evaluation
• 3 groups of REF evaluators
1. Output only evaluators (n (interview) =8)
2. Impact only evaluators (n (interview) =9); AND
3. Output and Impact evaluators (n (interview) =47)
Brunel University London
23 September 2014
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 2

3. The UK REF2014 Evaluation process 23 September 2014
Brunel University London
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 3
MAIN PANEL A
Sub-panel 1 – Clinical Medicine
Sub-panel 2 – Public Health, Health
services and Primary care
Sub-panel 3 – Allied Health Professions,
Dentistry, Nursing and Pharmacy
Sub-panel 4 – Psychology, Psychiatry and
Neuroscience
Sub-panel 5 – Biological Sciences
Sub-panel 6 – Agriculture, Veterinary
and Food Science
Evaluation items
Outputs – 50,317
Impact case studies
– 1,621

4. Brunel University London
23 September 2014
2
High significance
High Reach
(4*)
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 4
Significance
Reach
1
High significance
Low Reach
(¿2* or 3*?)
4
Low significance
High Reach
(¿2* or 3*?)
3
Low significance
Low Reach
(¿0-1*?)

5. Methodology – The Treatment 23 September 2014
Brunel University London
5
• Semi structured pre-evaluation interviews with Main Panel A and its sub panel
evaluators.
• Analysed using cognitive-based grounded theory
• Questions asked about the evaluators background (most proud), opinion of
impact (definition, what is important), and strategies for overcoming barriers in
evaluation (panel roles, attribution, definition differences etc)
EVALUATION PROCESS
Pre-evaluation
Interviews
Post-evaluation
Interviews
Jan-Mar 2014
Dec 2014 – Feb 2015
(1) Definitions
(2) Opinions
(3) Strategies
(4) Intentions
(5) Biases
(1) Re-test (1)-(5)
(2) Process
(3) Conflicts
(4) Power roles
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators

6. Main Panel A and its subpanels 23 September 2014
Brunel University London
6
Sub-panel name
No. of
sub-panel
members
No. of academic
evaluators (AEs)
(% AE’s / sub-panel)
No. of research
user evaluators
(UEs)
(% UEs/ sub-panel)
No. of
participants
Main Panel 19 14 (73.7%) 5 (26.3%) 8 (42.1%)
Sub-panel 1 – Clinical
Medicine
39 32 (82.0%) 7 (18.0%) 10 (25.6%)
Sub-panel 2 – Public
Health, Health services
& Primary care
27 23 (85.1%) 4 (14.9%) 13 (48.1%)
Sub-panel 3 – Allied
Health Professions,
Dentistry, Nursing &
Pharmacy
51 42 (82.3%) 9 (17.7%) 14 (27.5%)
Sub-panel 4 –
Psychology, Psychiatry
& Neuroscience
35 28 (80.0%) 7 (20.0%) 9 (25.7%)
Sub-panel 5 –
Biological Sciences
35 30 (85.7%) 5 (14.3%) 6 (17.1%)
Sub-panel 6 –
Agriculture, Veterinary
and Food Science
29 16 (55.1%) 13 (4.9%) 4 (13.8%)
TOTAL 235 185 (78.7%) 50 (23.2%) 62 (28.8%)
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators

7. Results
Pre-treatment – evaluation without precedence
• Uncertainty of Impact evaluation
LACK OF DEFINITION
• “I´m still not convinced everybody shares exactly the same definition of what constitutes impact or
where they place the weight or if it’s impact or isn’t” P3Imp1
LACK OF EXPERIENCE
• “I’m very happy to describe the quality of the research [but] the valuing of impact is something I
have no idea about” P0P2 Out-Imp1
• Resort to evaluation “comfort zone” (“what we cut our teeth on”)
USE TRADITIONAL TOOLS FOR RESEARCH EVALUATION RATHER THAN “impact stuff”
“And I don’t believe that we know how to do it- you have to contrast this with the
assessment of outputs which is really just reviewing, which is bread and butter stuff for an
academic. That’s what we cut our teeth on, that’s what we do every day and so there may
be an awful lot of it…but it is just what we do. Whereas this impact stuff we just don’t know.
So I feel a little bit nervous about it.” (P0 P2 outimp1)
Brunel University London
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 7

8. Results
Experience with ex-ante impact evaluation
• Previous experience with RCUK “pathways to impact statements” – potential
future impact
NOT FORMAL, A “tick box criteria”
“The research council introduced this criteria, it's just a tick box. But it's changing, it's a slow process,
you can't instantly get scientists to change their view. So they got this box, you may just tick it. We
will tell them why this will have amazing impact on humanity for the rest of eternity, and everybody
ticks that and then the REF comes along…. “ P2 OutImp 5
REGARDED AS A “DEAD WEIGHT”
“But that sometimes becomes such a dead weight around the necks of the people making the
decisions that it outweighs everything else, including those other sciences that could help.” P2Imp1
EX-POST EVALUATION FOR REF2014 ONE “big experiment”
“I think it's all new territory for all of us, and none of us know – we are going to learn on the job I
think.” P4OutImp6
Brunel University London
23 September 2014
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 8

9. The balance scale of impact evaluation 23 September 2014
Quality focused Impact focused
Brunel University London
Tendencies /
Decisions
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 9

10. The “societal impact” focused evaluator
• Impact was regarded as independent of the research underpinning it.
“The quality of the research has zero role at all in ensuring the impact of the research’”
P6 OutImp 2
“The research that is at application is not necessarily getting in these journals but it
could be very important to get something into the marketplace.” P0 OutImp3
• Not pre-occupied with the underpinning research
“What maybe a product or an end result of research has different criteria associated
with it because what you’re looking for here is a societal change…..…whereas the
research….…it’s quite different……[it]…..is all around rigor and methodology and the
quality of the idea and making sure that the methods and the quality of the idea match”
P2 OutImp7
Brunel University London
23 September 2014
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 10

11. Tendencies of the societal-impact focused
evaluator
• DECISION 1 – Impact is serendipidous / uncontrollable / Messy
• “‘it can often be that coffee you have with somebody at the right moment and the information
passing that way’” P1Imp1
• “That kind of linear approach is very, very rare indeed. The way it is instead is that findings
accumulate over a period of time and either the weight of evidence in the end wins the day or a
moment arrives when the politicians have made up their minds that they want to go in a particular
direction, they are looking around for the evidence to support their decision “ P2 Imp1
• DECISION 2 – Value of push factors in achieving impact
• “You can’t assume that it will happen through happenstance, there needs to be some mechanism
in place.” P1Imp1
• “Getting from the university stage of research out to the end impact has so many steps in it, not
all of which are easy. They require a little effort and somebody championing them from one end or
the other” P1 OutImp 4
Brunel University London
23 September 2014
Presentation Title 11

12. Tendencies of the societal-impact focused
evaluator cont.
• DECISION 3 – Hard vs soft impacts (Impact outcome vs impact journey)
• “If you think of impact as a verb rather than a noun, I think it’s a lot easier to analyse. Impact is
the relationships you build. It is the dialog that you have that makes you ask research questions
that are subtly different from the ones you would have asked if you hadn’t linked with - whether
it's policymakers, whether it's citizens, whether it's industry at the beginning. So impact is not
something that you have right at the end. Impact is a relationship and that attitude of mind that
you have throughout the research process” P0 OutImp4
• ‘levering it to the next stage’: [whatever] gets the research being taken up and moving it forward,
that has to be considered valuable. Maybe the question we should be asking is whether enough
effort has gone into that in the past and levering research into its next stage” P0 OutImp6.
• DECISION 4 – Can it be measured?
• “And it’s important not to ignore them because you can’t measure them because sometimes you
throw out the most important things because you can’t measure them properly.” P1 OutImp2
• “These [soft] are unquantifiable, so therefore it will be difficult to assess them as impact. And
again, it’s subject to imagination, you don’t know how you’ve affected anything until you see the
results. So the only time that you know there is an impact is when there is a result. So therefore
just talking to people is not an impact." P6 Out2
Brunel University London
23 September 2014
Presentation Title 12

13. The “quality” focused evaluator
• Preoccupation with considering the underpinning research as a proxy for impact
• “I think research will only have an impact if it’s of high quality. I think that quality is the sine qua
non of impact” P0 OutImp2
• “So you can get big impacts from very, very bad quality of research, and so that's -- if that's the
way you are going to measure impact then you're going to go completely the wrong way.” P0
OutImp2
• Belief in the mantra of excellent impact being dependent on excellent research
• “I think that certainly the quality of the research is an important part. It’s a critical part. You have
to have the highest quality research in order for it to be believable and repeatable.” P0 OutImp5
• Bias towards “applied research” as “easy impact”
• “It's [her applied work] is not the research I've done that's the most prestigious in terms of, if you
like, the judgment of academia for I've done other research that have been associated with bigger
grants, research council grants and if you like higher impact academic products. But [this applied
work] is the easiest to demonstrate a real impact.” P3 OutImp2
• “And that the impact case studies might be based on, you know, research is good enough, the
equivalent of a two-star, but it won't be four-star research, and but it has an impact because it was
applied research.” P2 OutImp 2
Brunel University London
23 September 2014
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 13

14. Tendencies of the quality-focused evaluator
• DECISION 1 – Research excellence and impact
DANGER OF IMPACT OF BAD RESEARCH
• “It can’t be actually used or believed until its repeated and proven.” P0 OutImp 5
• “I think that poor quality research can only have negative impacts.” P4 OutImp8
HOPE THAT ASSUMPTION OF VALUE OF EXCELLENT RESEARCH IS CORRECT
• “ ….excellent impact should depend on excellent research” P1 OutImp6
• “…you would hope they were synonymous, wouldn’t you?” P3 OutImp 5
• DECISION 2 – Linearity of impact realisation
RATIONALISATION OF IMPACT REALISATION
• “Research was done, showed the benefits of [the evidence], got into the clinical guidelines, and
over time you can track the proportion of the relevant professionals who are implementing the
better evidence. It’s quite straightforward in fact” P3 OutImp8
• “Impact requires that you generate the evidence and then that you, in turn you get into guidelines
and the people start using that information to change their practice’” P4 Out1.
Brunel University London
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 14

15. Tendencies of the quality-focused evaluator
cont.
• DECISION 3 – Pull / Picked up factors
SOMEONE ELSE’s JOB – RESEARCHERS JUST DISSEMINATE
“I mean, I presented this work at the International Embryo Transfer Society meeting, and then
afterwards, some people talked to me and what should we do. And I said, well, you’ve just got to
purify these hormones better and you have to do this and that and they did it. Then it turned out to be
very important for a company called Bioniche because they now have something like 80, 90 percent
of the market for the hormones used in superovulation.” P0 OutImp1
• DECISION 4 – Eventually all excellent research has impact
LINK BETWEEN RESEARCH QUALITY AND IMPACT AS WELL AS “ALL IN GOOD TIME”
“….my feeling is that eventually it will [have impact] but it may take a long time.”
• DECISION 5 – Assessing the “right impact” / Lesson from MMR
DANGER OF EVALUATING IMPACT TOO SOON
“The example…is the MMR story, that was poor science. It’s had huge impact, negative impact. Its
resulted in lots of morbidity amongst children plus women -- people didn’t get their children
vaccinated. The quality of the science was poor, but it had a huge impact in a negative way.” P2
OutImp9
Brunel University London
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 15

16. Discussion – what this means for Impact
evaluation
• Is there a bias? Depends on definition of bias? Perhaps a conservative bias.
• REF2014 showed a variety of values about what constitutes impact, what will be
assessed as part of impact.
• Some of these went beyond the HEFCE or REF2014 definition of impact eg. Impact
journey valued, as was the role of public engagement.
• Concern that evaluator´s lack of experience in impact evaluation might do one of 2
things
1. Force them to use traditional measures as proxies for impact (Quality focused evaluator) but also
any type of quantitative indicators eg. QALY, deaths/lives saved, % GDP, $$$$
2. Make haphazard judgements about impact being absolute (eg. Give everything 4* because
Impact is impact) – more Societal impact evaluator.
• What will tip the scales will be worked out during the evaluation process “…learn on
the job.”
• Post evaluation interviews will serve to investigate this further.
Brunel University London
23 September 2014
Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators 16