Societal Impact Nicolas Robinson Garcia, INGENIO (UPV-CSIC), Universitat Politècnica de València, Spain / Daniel Torres-Salinas, Universidad de Navarra and Universidad de Granada (EC3metrics & Medialab UGR), Spain Recently there is an increasing pressure on the development of indicators and methodologies that can offer evidences of the societal impact of researchers’ activity. This presentation will offer a comprehensive overview on the definition of societal impact, types of impact, and the attribution problem when searching for potential indicators. A special attention will be given to altmetric indicators and their potential role in tracing social engagement and its relation with societal impact. Examples of potential uses and current lines of work will be presented. *************************** Scientometric procedures are increasingly used to analyse developments and trends in science and technology. Decisions to be taken often have severe implications. Consequently data handling, indicator construction and interpretation require competent expert knowledge, which is currently only available to a limited extent for all stakeholders in Central Europe not the least due to lacking training opportunities. Responding to the lack of a pertinent scientometrics education (especially in German speaking countries) and to the increasing demand (particularly of research quality managers), the University of Vienna (A), the German Centre for Higher Education Research and Science Studies - DZHW (D) and the Katholieke Universiteit Leuven (B) joined cooperatively to found the European Summer School for Scientometrics (esss) in 2010.

1. European Summer School| September 8, 2016
Societal Impact
Nicolas Robinson-Garcia Daniel Torres-Salinas
INGENIO (CSIC-UPV) Univ Navarra and Univ Granada
(ec3metrics & Medialab ugr)
www.ingenio.upv.es

2. Objectives
Understand the motivations that lead for
demands of societal impact indicators
1
Know the limitations and current problems
existing to meet such demands
2
Learn about potential uses of altmetrics to
showcase ‘aspects’ of societal impact
3

3. Index
Brief Intro
What is societal impact?
How can we measure societal impact?
Altmetrics and the ‘broader impacts’ of research
Potential uses of altmetrics
Food for thought

4. “The last 30 years have witnessed a slow, but
continuous process of reorganization of European
universities”
Geuna & Muscia, 2009
Brief intro
https://static.pexels.com/photos/30824/pexels-photo.jpg

5. Brief intro
Global competition and budget cuts
Accountability and transparency
Evaluation and funding

6. Brief intro
• Bibliometrics is seen as an answer to provide impact
measures within and even outside the scientific sphere
citation analysis – patent analysis – univ-industry collab
• In the recent years there has been a critical revision on
the limitations of these metrics
scientific impact ≠ societal impact
http://www.logic-canvas.com/wp-content/uploads/2013/11/Transparency-Glass.jpg

7. Brief intro
DORA Declaration
Against the use of journal based
indicators
Leiden Manifesto
Best practices on the use of
bibliometric indicators
The Metric Tide
UK Report on the role of bibliometric
indicators in research evaluation

8. What is societal impact?
Societal impact is usually referred as the
social, environmental, cultural or economic
benefits derived from academic activities
Teaching Research
Third
Mission
SOCIETAL IMPACT

9. What is societal impact?
Type of impact Output Activities
Scientific-
scholarly
Knowledge Growth Papers, books, reports,
cases, magazines
Increase and
organise the stock
of knowledgePublication outlets
Research networks
Workshops,
collaboration,
conferences,
committees
Build and convene
networks
Supply of human
capital
Societal
Cultural
Archives, museums,
festivals, exhibition
and performance,
events
Social Policy changes, social
benefits
Problem solving
Socioeconomic Patents, spin-offs,
start-ups, contracts,
consultancy
Support
entrepreneurialismTechnological
Source: Robinson-Garcia, van Leeuwen & Rafols, in progress

10. How can we measure
societal impact?
Socioeconomic & Technological Impact
• Largest body of literature
• Synergies between industry and university
• Tracking of knowledge transfer
Social Impact
• Translational research
• Environmental research
Cultural Impact
• Field-specific
• Largely unexplored
• Affects mainly the Social Sciences & Humanities

11. How can we measure
societal impact?
The attribution problem
Researcher
publishes a paper
Researcher gives
lectures?
Media coverage?
Patent protection?
Advocacy?
Local council
implements new
policy
Increase on
tourism rates
Employment rates
increase in industry
Creates a spin-off?

12. How can we measure
societal impact?
The attribution problem
Researcher
publishes a paper
undertakes a
research project?
Local council
implements new
policy
Researcher gives
lectures?
Media coverage?
Patent protection?
Advocacy?
Increase on
tourism rates
Employment rates
increase in industry
Creates a spin-off?

13. How can we measure
societal impact?
An example. El Cabanyal
The district
•Old village, now a district
•Architectonic value
The sting
•Access to the beach
•Salvem el Cabanyal
The link?
•Few papers (National journals)
•Not all researchers involved are authors
The attribution problem

14. GOALS
How can we measure
societal impact? UK REF
Selective allocation of research funding to Higher
Education Institutions
Produce evidence of benefits of public investment in
research
Provide benchmarking information and reputational
yardsticks
King’s College London & Digital Science, 2015

15. First attempt to allocate funding based on non-
academic impact
How can we measure
societal impact? UK REF
King’s College London & Digital Science, 2015
“Impact is defined as ‘any effect on, change or
benefit to the economy, society, culture, public
policy or services, health, the environment or
quality of life, beyond academia’ (REF, 2011)”

16. How can we measure
societal impact? UK REF
PANEL A
Biomedical & Health Sciences
6 UoA – 1586 case studies
PANEL D
Humanities & Arts
10 UoA – 1617 case studies
PANEL C
Social Sciences
10 UoA – 1965 case studies
PANEL B
Life & Earth Sciences,
Engineering, Physics, Maths
9 UoA – 1469 case studies

17. Assessing societal impact through impact case
studies
How can we measure
societal impact? UK REF
IMPACT CASE STUDY TEMPLATE
1. Summary of the impact
2. Description of the underpinning research
3. References of the research
4. Details of the impact
5. Sources to corroborate the impact

18. How can we measure
societal impact? UK REF
Case studies by type of impact

19. How can we measure
societal impact? UK REF
Case studies by type of impact

20. How can we measure
societal impact? UK REF
Ratings by type of impact

21. How can we measure
societal impact? UK REF
Wilsdon et al., 2015
“[…] for the impact component
of the REF, it is not currently
feasible to use quantitative
indicators in place of narrative
impact case studies, or the
impact template”

22. Altmetrics and the ‘broader’
impacts of science
Wilsdon et al., 2015
“[…] wider use of quantitative
indicators, and the emergence of
alternative metrics for societal
impact, could support the
transition to a more open,
accountable and outward-facing
research system”

23. Altmetrics and the ‘broader’
impacts of science
Classic perspective Author perspective
Two perspectives

24. Case: societal impact
indicators at UGR
Multidimensional evaluation using biplot.
Research at the University of Granada
Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis
at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de
Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis
Research questions:
• ¿What is the metric
profile of research
groups taking into
account bibliometric
and non-bibliometric
indicators?
Methodology:
• Time frame: 2009-2013
• Analysis unit:
161 research groups = 2216 res.
• Methods:
Correlations
biplot analysis

25. Case: societal impact
indicators at UGR
We calculated
46 indicators
and classified them into
9 different dimensions
We reduced and make a biplot analysis
Multidimensional evaluation using biplot.
Research at the University of Granada

26. Case: societal impact
indicators at UGR
Altmetrics
Funding and projects
Scientific output
Knowledge transfer
Media and news impact
Scientific impact – Citations
Teaching dissemination
9 Dimensions of academic
activity and impact

27. Case: societal impact
indicators at UGR
Altmetrics
Funding and projects
Scientific output
Knowledge transfer
Media and news impact
Scientific impact – Citations
Teaching dissemination
Societal impact related dimensions

28. Case: societal impact
indicators at UGR
Altmetrics
Google +
Mentions
Facebook
Likes
Twitter
Indicators
Knowledge
Transfer
Patents
Contracts
Spin offs
Media and
News Impact
Newspapers
contributions
Public
exposure
Mentions in
Newspapers
Teaching
Dissemination
Teaching
Material
Innovation
Projects
Some examples of “societal impact” indicators

29. Case: societal impact
indicators at UGR
Some examples of the information sources

30. Case: societal impact
indicators at UGR
Method remarks
1) Calculation of 46 indicators for research
groups in four areas
2) Normalization and standarization of
indicators values using z-scores
3) Representation of dimensions and groups
into two dimensions using biplot analysis
4) Reducing the biplot with clockwise
representation
Now we are going to see an example in
Exact Sciences

31. Potential applications
of altmetrics
These are the
dimensions
(blue)
These are the
research groups
(green)
BIPLOT: “EXACT SCIENCES 52” Research groups

32. Potential applications
of altmetrics
Teaching Dissemination
Media and News Impact
Knowledge Transfers
Altmetrics These are cluster and
dimensions used to identify
SOCIETAL IMPACT

33. Potential applications
of altmetrics
Altmetrics
What we see is that most of the research
groups are in dimensions related to
traditional publications and the impact
system (ie: Web of Science)
Scientific Impact
Scientific output

34. Potential applications
of altmetrics
Altmetrics
DOES ALTMETRICS
MEASURE
SOCIETAL IMPACT?
We see that altmetrics is
more related to scientific
impact and scientific output
(at the UGR)
Scientific Impact
Scientific output

35. Potential applications
of altmetrics
Teaching Dissemination
Media and News Impact
Knowledge Transfers
So, for Exact Sciences at the University of
Granada we have a really clear quadrant
related to SOCIETAL IMPACT

36. Potential applications
of altmetrics
We have identified four groups with a
clear Societal Impact orientation
FQM325
FQM171
FQM338
FQM325
FQM324

37. Teaching dissemination
Media and news impact
Funding and projects
National scientific output
Knowledge
transfer
Scientific impact - Citations
International scientific output
Altmetrics and webmetrics
Finally this is the general and multidimensional
overview of the exact sciences at the UGR

38. Potential applications
of altmetrics
Social
engagement
through Twitter
1.Select a researcher
2.Identify its follower/followee network
3.Identify different communities of
contacts
4.Identify geographical/cognitive/…
proximities

39. Potential applications
of altmetrics
Social
engagement
through Twitter
L Waltman’s Twitter
network

40. Potential applications
of altmetrics
L Waltman’s Twitter
network
Social
engagement
through Twitter

41. • Threats derived from a purely qualitative
assessment
Peripheries and prestige
• Threats derived from purely quantitative
assessment
Biases by type and scope of impacts
Visualizations as a balanced solution
Food for thought

42. European Summer School | September 8, 2016
Questions?
Nicolas Robinson-Garcia Daniel Torres-Salinas
INGENIO (CSIC-UPV) EC3Metrics SL
www.ingenio.upv.es

43. European Summer School | September 8, 2016
Especial thanks to Yusnelkis Milanés
for providing data from her PhD thesis
www.ingenio.upv.es
Source: Milanes-Guisado, Y. (2015). Multidimensional evaluation of research. Micro analysis
at the university of Granada during the period 2009-2013 [doctoral thesis]. Universidad de
Granada. Supervisors: Daniel Torres-Salinas, José Navarrete, Francisco Solis