SlideShare a Scribd company logo
1 of 63
Download to read offline
INDICATORSOFQUALITY??
OUTLINE
INDICATORS OF QUALITY??
• Methods of quantitative evaluation
• University level – Macro bibliometric analysis
• Individual level – Micro bibliometric analysis
Lorna Wildgaard
Royal School of Library and Information Science
Lorna.wildgaard@hum.ku.dk
Kildekritik, F2016
STUDY OF QUANTITATIVE ASPECTS OF:
informetrics
bibliometrics
scientometrics
webometrics
cybermetrics
© Björneborn
altmetrics
Information in any
form by any social
group
Production,
dissemination & use of
recorded information
Science as a discipline
or economic activity
Politico-economical
aspects.
Construction & use of
information resources &
tech. on the internet
Web documents, text &
multi-media, blogs, tags,
wikis..
Dissemination, use & online
activity of scholarly content
SO METRICS ARE:
”The application of mathematical and statistical
methods to books and other media of
communication.”
Pritchard, A (1969), Journal of Documentation, 25(4):348-9
e=18
h=8
m=1
𝑿=3 CPP=18
t=4
PERCENT DISCIPLINARY COLLABORATION
C=
Nm
Nm+Ns
Nm = Number of multi-authored papers in the
discipline, eg. 22929
Ns=Number of single authored papers in the
discipline, eg. 9325
C=
22929
22929+9325
= 0.71 * 100 = 71%
S c (i) = γ ∗ (Y (now) − Y (i) + 1)−δ ∗ |C(i)|
Weighted
citation score
of an article
Actual Year
minus
publication year
Citation count
Coefficient set at 4, i.e. for an article published
during the current year, its citations account
four times
WHAT IS A PUBLICATION?
A scholarly publication is a an expression of the
state of the researcher at a particular time (Price
1970)….the product of scientific research (Cawkell
1976; Lazarev 1996)….what consitutes a publication
is different in the scientific disciplines (Castellini
2014; Hicks 2012; Tinkler 2011; Frandsen &
Nicolaisen 2011)….different publication types
indicate different methodologies (Grant & Booth
2009;Dzombak 2013)….Publications should be
conceptualized as a discrete set of objects so they
can be counted in aggregate (Skupin 2009)….output
in a citation index (Antonakis & Lalive (2008).
What constitutes a ”publication” should be clearly
defined to ensure representative operationalization
in the indicator and the extraction of meaningful
relationships
(Wouters 1999; Skupin 2009; Colledge 2012)
WHAT IS A CITATION?
Citations reflect som sort of cognitive influence (Moed
2005)….the strength of the influence is not known (Martyn
1964)….income or reward (Ravetz 1971)….meaning of citation is
governed by paradigmatic & social norms (MacRoberts &
MacRoberts 1996; Cronin 2000)….acknowledgement of
intellecual debt to other works (Merton 1971)…. Markers or
symbols of influence in the aggregate (Small
1978;1987)….citation impact (Martin & Irvine 1983)….rhetorical
devices (Luukkonen 1997; Latour 1987)….authority (Gilbert)
quality sensor mechanisms (Lindsey 1978)….A manipulable
measure of something or other (Singleton 1976)….influence on
average (Nicolaisen 2004)…citations are arbitrary (Leimu &
Korcheva 2005)… markers (footprints) of scientific
communication (Garfield 1979)….income (Albarrán 2011)……
The concept of citation is open for interpretation,
of which there are numerous, and it is impossible
to exclusively link citations to quality
WHAT IS A RESEARCHER?
Dias 10 21/02/2016
WARM UP EXERCISE
https://todaysmeet.com/Warm_Up
Handout: Researcher evaluation
1. Suggest indicators that could measure
attributes, skills, interests and
achievements.
2. Consider how to indicate the quality of
these attributes, skills, interests &
achievements.
EVALUATION
OUTPUT
DATA
MODEL
MOTIVATION TO PUBLISH
MOTIVATION TO CITE
INTERPRETATION OF ALI
LEGITIMATE LINK TO REAL
WORLD
PERFORMANCE OF RESEARCHER
IN SYSTEM REDEFINES WHAT
SUCCESS IS
EXOGENOUS VARIABLES
WHAT IS MISSING
INDEXING POLICY
THE PROBLEM
COFFEE BREAK: Meet back here in 10 minutes
UNIVERSITY LEVEL
RESEARCH PERFORMANCE
EVALUATION SYSTEMS
-Register the amount & type of a scholar’s publications.
REF (UK): Panel-based evaluation every 6th year.
Quality=peer review.
BFO (Flanders): Citation-based
Quality=reception & use
ERA (Aus)=Publication-based (udifferentiated).
The number of publications dictates the amount of funds given to
the university & the assessment of a
researcher’s productivity.
Quality= not assessed
Norwegian indicator 2004 (differentiated)
Tiered system of publication types and sources.
Aim: increase publications in international journals and establish a registration system in
which institutes can validate & verify publications as well as see other institutions
publications
Quality=not addressed
BACKGROUND FOR BFI
BFI (DK) 2009-2012 (differentiated)
Tiered system of publication types and sources
Aim: Distribute funding based on the quality of research results
(UFM 2015a). Universities are judged based on published research.
Quality= explicit assumption: the journal's status is assumed to say something about the
individual publication’s quality (UFM 2015a). 16
BFI Norge
Quality indicator Publications indicator
25% of a sub-pool of basic funds,
amount changes each year
2% of all university funds each year
Domain neutral Normalised. Benchmarked against total
publication points and domain share
Unclear, complicated Clear, simple
Publication performance Stimulate publication in international
resources
Accountability model, aimed at
distributing funds, production
measured in Danish Crowns and
legitimacy
Registration model
Fractionalised count ”Whole” counting
International/eksternal collaboration
rewarded with more points
Points normalised to the domain w.r.t.
speed of publication, collaboration
traditions and publication type
Book publishers on level 1 & 2 Book publishers on Level 1
TEACHING
40%
FACULTY
WORK
23%
ADMIN.
18%
RESEARCH &
PUBLICATION
12%
NETWORK
7%
WHAT IS THE BFI MEASURING?
HOW
RESEARCHERS
SPEND THEIR
TIME: WEEKLY
AVERAGE
Ziker (2014) https://thebluereview.org/faculty-time-allocation/18
BFI
GROUP AREA FIELD COVERAGE
1-19 HUM
Language, Culture, History, Linguistics & Semiotics, Literature,
Dance, Music, Theater, Art History And Art, Cultural Studies, Media
& Communication, Film, Rhetoric, Theology, Philosophy,
Archeology, Pedagogy, Education, Anthropology, IT & Humanities
20-27 SOC. SCI
Law, Sociology, Social Work, Political Science, Geography, Social
Economics, Psychology, Library & Information Science
28-46 SCI/TEK
Geo-physics, Physics, Astronomy, Math, Chemistry, Biology,
Environmental Science, Agriculture, Molecular Science,
Pharmacology, Civil Engineering, Transport, Electronics, Energy,
Chemical Engineering, Nanoscience, Medical Technology, System
Development And Entrepreneurship, Food Science
47-62 SUND
Medicine, Surgery, Anesthesiology, Neurology, Ear-nose-throat, Eye
Disease, Gynecology, Dermatology, Pediatrics, Pharmacology,
Dentistry, Nursing, Public Health, Veterinary, Pathology, General
Medicine
63-68 OTHER
Gender Studies, Architecture, Town Planning, Sport, Digital
Communication, Information Systems & IT Management, Science
Studies And Research Analysis
19
BFI STRUCTURE: PUBLICATION TYPES
Publication types N. 1 N.2
Monography in ISBN title (inc. reports, ex. Text books) 5 8
Monography in book series 5 8
Contribution to anthology, ISBN title (inc. introductory & final chapters;
book chapters; ex., abstracts, lecture manuscripts, PPTs, encyclopedia,
preface/afterword, commentaries, editorial work)
0.5 2
Contribution to anthology, bookseries 1 3
Contribution to a conference, series 1 3
Contribution to a conference 0.5 2
Article (inc. letter, review ex., abstracts, editorial letter, commentary,
book review)
1 3
Doctorate 5
Patent 1
Publication types registered in PURE, PURE4 42 basic publishing types:
http://bibliotek.science.ku.dk/forskdok/docs/metadatamodel_P4_LIFE_GR_270312.pdf20
CALCULATION OF BFI POINTS
Points are shared between universities dependent on how many
authors belonging to the university are listed on the publication:
A book on level 2 gives 8 points. There are 3 authors from AU, 2 from
CPU & 1 from a foreign university.
AU = 3/6 points
CPU=2/6 points
8/6 = 1.3 points to each author
Collaboration with authors from other universities, Danish or foreign,
is rewarded, The total number of points are multiplied by 1.25.
AU total points= 1.3 * 3 =3.9 * 1.25 = 4.87 points
CPU total points = 1.3 * 2 =2.6 * 1.25 = 3.25 points
BFI is domain neutral (funds are shared within domain
and not across domains)
21
DISTRIBUTION KEY
(fordelingsnøglen)
2009
45% STÅ INCOME
20%
EKSTERNAL
10%
PhD
25% BFI
2015
2008
50% STÅ INCOME 40% EKSTERNAL
10%
PhD
2000
Tax catalogue: http://ufm.dk/uddannelse-og-institutioner/videregaende-
uddannelse/universiteter/okonomi/uddannelsesbevillinger 22
WHAT HAPPENS AFTER 2015?
BFI must only distribute new funds and not alter the existing
taxometer for distribution of basic university funds (DU 2009)
• New funds = globalization pool
• The globalization pool is expected to be emptied in 2015
• Suggested 25% BFI distributions parametre is supplemented
by taking funds from external grants
• Basis funding is grouped together with other funding and
therefore effects distribution and share. fx:
HUM greatest operational costs, greatest number of students,
lowest STÅ tax and lowest amount of funds invested in
researcher education, researcher support, administration and
traditionally the lowest number of publications.
http://www.altinget.dk/forskning/artikel/basismidler-fylder-
mindre-paa-universiteterne 23
COLLABORATION PARAMETER
Publications
Level
Total Points
%
National
points
Collaborative
publications
% Collaborative
publications
1 2
NAT/TEK 2921 1596 1325 2973.99 24 2406 82
SUND 3437 2379 1058 2851.26 42 3056 88
HUM 963 625 338 1502.72 34 185 19
SAM 739 448 291 1141.47 21 225 30
I ALT 8060 5048 3012 8469.44 30% 5872 72%
*Universities share of the total number of national points, given in percent
https://bfi.fi.dk/Publication/NationalAnalysis
24
PUBLICATION POINTS (UFM 2015)
% diff. i p.p 2013-2014 % diff. antal p. 2013-2014
Aalborg U +35.3 +37
Aarhus U +4.1 +2.6
CBS +20.2 +11.6
DTU -0.3 +3.7
ITU +19.5 +12.5
KU +0.3 -3.7
RUC +7.7 -4.9
SDU +16.1 -16.4
DOMAIN % diff. 2013-2014
NAT/TEK -0.5
SUND +16.9
HUM +15.3
SAMF +12.5
25
EVALUATION OF BFI
Good focus on strategic publishing. BFI works as an
incitement to publish in certain resources.
Yet, the development of longterm, dialogue based
communication forms with public society could be
ignored in a quest to fulfill the requirement of selling
science through an unidirectional intemediatory
(red.: journals).
(Schneider, 2015)
26
VISIBILITY OF HUM IN CITATION INDICES
0
50
100
150
200
250
300
350
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Numberofpublications
Publication year
HUM i WOS
HUM i Scopus
BFI
P=963 BFI
P=813 BFI
27
EVALUATION OF BFI
UFM(2015)
1. BFI lacks transparancy
2. Need for better communication between
stakeholders
3. Citation indicators could be advantages in some
domains. This will enable analysis of
international competition.
The performance of BFI in the model to distribute
funds was not assessed
28
CONSIDERATIONS
BFI results in an increase of research publications in
level 1 and 2 sources, and increased collaboration
(Ingwersen & Larsen, 2014)
BFI results in an increased attention on the
researcher
employment
promotion
funding applications
Under contract to publish in BFI resources and
produce publication points(AAU Bestyrelse, 2014;
FOU., 2013; Strandskov, 2011)
29
LOOKING TO THE FUTURE…
Does BFI prioritize what is measurable over what
should be measured but is difficult to quanitify?
(Aagaard 2012)
• Leiden manifesto:
http://www.leidenmanifesto.org/
• Snowball Metrics:
http://www.snowballmetrics.com/
• Responsible Metrics
https://responsiblemetrics.org/
COFFEE BREAK: Meet back here in 10 minutes
INDIVIDUAL LEVEL SCIENTOMETRICS
TYPES OF INDICATORS
Hybrid
metrics
PUBLICATION METRICS
count production,
collaboration, &
contribution
CITATION METRICS
count citations, self-
citations, adjust for
collaboration
Table of indicators: http://tinyurl.com/nj4mvca
Ranking, growth, proxy for excellence, effect,
field comparisons & quality
INTERNAL CHARACTERISTICS OF ALI
Wildgaard et al (2014) A review of
the characteristics of 108 author-
level indicators. Scientometrics,
101(1):125-158
Age: years since first publication (WoS) = 2016-2009 = 7
Papers: all publications (WoS)=13
Citations: total citations (WoS)= 190
1st subject area=Anesthesiology
2nd subject area=Neuroscience
3rd subject area=Surgery
INDEX OF QUALITY AND PRODUCTIVITY
http://tinyurl.com/zlxzyz9
Antonakis, J. and Lalive, R. (2008), Quantifying Scholarly Impact:
IQp Versus the Hirsch h. J. Am. Soc. Inf. Sci., 59: 956–969.
doi: 10.1002/asi.20802
http://www.hec.unil.ch/jantonakis/IQp%20calculator%20version
%202008.xls
INDEX OF QUALITY AND PRODUCTIVITY
190
Cited 1.44 times more
than average paper in
the specialty
Number of papers
performing above
average for the specialty
JIF of 3 subject
categories
ANOTHER ASPECT OF QUALITY?
WEB/ALTMETRICS
WEB 2.0 = PARTICIPATIVE WEB
”… supports, extends, or derives added value from
human social behaviour …”
TWO ELEMENTS IN NETWORKS
Node
Nodes represent things
that relate somehow to
one another
Tie
Tie represents relations
between nodes
© James Cook
https://www.youtube.com/channel/UCerAw4EfTOnYYxLLPZAzMxQ
TWO KINDS OF NETWORKS
DIRECTED NETWORKS
Networks in which the tie has a
driection (digraphs), eg. Linking to
a webpage, downloading a
document, citation networks
UNDIRECTED NETWORKS
Networks in which the tie has
no direction eg. friends on
Facebook,
© James Cook
INDUCED HOMOPHILY
Tendency for ties to form
similar others because
similar others present the
social environments
(group, community,
society)
No wonder blues are mostly
tied to blues…there are
hardly any reds out there! © James Cook
CHOICE HOMOPHILY
A tendency to choose to form ties
with similar others even when
different others are available in
the social environment (group,
community, society)
© James Cook
PRINCIPLES OF NETWORKS
• Ties should be straight
• Ties should not be far from each
other
• Ties should not cross or touch
• Ties should be easy to follow
from node to node
• Nodes that connect should be
close
• Similar nodes should be close to
one another
• Central nodes should be in the
center
© James Cook
http://noduslabs.com/research/knowledge-graphs-type-document/
LINKED IN: CONNECTIVITY
Quick-Diverse-Broad
A record of attention: how many people have been exposed to
and engaged with a scholarly output in the news, blogs, and on
Twitter; article pageviews and downloads; GitHub repository
watchers.
A measure of dissemination: where and why a piece of research is
being discussed and shared, both among other scholars and in the
public sphere.
An indicator of influence and impact: Some altmetrics can signal
that research is changing a field of study, the public’s health, or
having any other number of tangible effects upon larger society,
eg. references in public policy documents; or commentary from
experts and practitioners.
An output
• Journal
Article
• Dataset
• Poster
An
identifier
attached to
the output
• DOI
• PubMedID
Mention in
a source
• Social media
• Blogs
• Wikipedia
• Media
API stands for Application Programming Interface
ALTMETRIC.COM
The colours of the donut each represent a
different source of information.
The attention score is a weighted count of all of
the attention a research output has received.
http://tinyurl.com/z7zozr5
Demographic breakdown Count %
Members of the public 153 18
Scientists 31 16
Science communicators (bloggers, journalists,
editors)
5 3
Practitioners (doctors, other healthcare
professionals)
3 2
Indicators are designed to measure particular
aspects of the effect of a researcher’s work – over
time, to field, as quality, ranking all or selected
works, co-authorship etc.
Judgements based on indicators can lead to
assumptions about the productivity and impact of a
researcher, which can be unsubstantiated, and
affect the psychological character of the individual.
ALI have to be methodologically sound
WHAT WE HAVE TO BE AWARE OF
51
ALI GIVE A SNAP-SHOT OF SELF
IMAGE AND CORE PERSONALITY
TRAITS
COMPARING RESEARCHERS
CAN EXPOSE THE
INDIVIDUAL
INDIVIDUALS USE ANY DATA TO
INCREASE ALI SCORES, TO INCREASE
THEIR SUBJECT VALIDITY & SELF-
WORTH
ALI BRING OBJECTIVITY TO THE
EVALUATION & REDUCE GENDER;
CULTURAL AND RACIAL BIAS
DOCUMENTING BEING OUT-
PERFORMED IS DETRIMENTAL TO
A RESEARCHER’S SELF-DEFINITION
ALI DO NOT ADD CONTEXT BUT
CAN ADD REDUNDANT
INFORMATION
SUCCESS IS DEFINED AS WELL
IN THE SYSTEM
EXTERNAL CHARACTERISTICS OF ALI
1. Transparency
Know which data is used to compute the indicator
Understand the math and the inferences
2. Demographics
Be aware if demographics affect the indicator
3. Motive
Indicators must fit the objectives of the evaluation
4. Diversity
Choose indicators that fit the discipline/publication
5. Openess
Make the limitations of indicator explicit, use
supplementary indicators
APPROPRIATE INDICATORS
“CAN WE ASSESS
THE BEAUTY OF THE
MONA LISA BY
COUNTING THE
NUMBER OF
VISITORS TO THE
LOUVRE?”
GROUP DISCUSSION. WRITE YOUR IDEAS AT
https://todaysmeet.com/Indicators_of_Quality
statistics 5,074,920
www.wikimindmap.org
http://en.wikipedia.org/wiki/Wikipedia:Statistics
http://stats.wikimedia.org/EN/TablesWikipediaEN.htm
STATISTICS up to 2009/2010
https://en.wikipedia.org/wiki/Wikipedia:Database_reports
REFERENCES
Aagaard, K & Mejlgaard, N (2012) Dansk forskningspolitik efter årstusinskiftet.
Aarhus Universitetsforlag
AAU (2013). Newsletter from the Rectorate, October 2013. Retrieved 23-6-2015
from: http://www.intern.aau.dk/ansatte/ledelsen-AAU/aau-
rektoratet/rektoratets-nyhedsbrev/vis-nyhedsbrev//nyt-fra-rektoratet-oktober-
2013---newsletter-from-the-rectorate-october-2013.cid97315#bfi
Barbulescu, Roxana. 2011. European Research Council’s Grant for Advanced
Investigators in Social and Political Sciences. European University Institute.
DS 2014: Svagtfald I statens forskningsbudget
(Nyt fra Danmarks statistik, 28 jan 2014)
http://www.dst.dk/pukora/epub/Nyt/2014/NR039_1.pdf
DU. (2009). Politikpapir om model for fordeling af basismidler. Faggruppe 68.
Retrieved 23-6-2015, from:
http://faggruppe68.pbworks.com/f/Politikpapir_170409__til_hjemmesiden_.pdf59
REFERENCES
DU. (2012). Overblik over finanslovforslaget for 2013. Retrieved 23-6-2015, from:
http://www.dkuni.dk/Politik/~/media/Files/Politiknotater/notat%20om%20ffl13%20(3
).ashx
Emmeche, C. (2014). Den Bibliometriske Forskningsindikator - fordele og ulemper.
Faggruppe68. Retrieved 23-6-2015, from:
http://faggruppe68.pbworks.com/w/page/6015700/Den%20Bibliometriske%20Forskn
ingsindikator%20-%20fordele%20og%20ulemper
FOU. (2013). Publiceringsstrategi for Rosklide Universitet 2015-2018. RUC Hearing.
Retrieved 23-6-2015, from: www.ruc.dk/?eID=minute&m=1948&t=179&docID=30986
Hansen, L. (2009) ‖Hvorfor er en humanistisk artikel mere værd end en
sundhedsvidenskabelig?‖ Indlæg på bloggen ‘Forskningsfrihed‘ 12/10-2009.
Ingwersen & Larsen (2014) Influence of a performance indicator on Danish research
production and citation impact 2000–12. Scientometrics, Online 19/4 2014; DOI
10.1007/s11192-014-1291-x.
60
REFERENCES
Meo, Sultan A., Abeer A. Al Masri, Adnan M. Usmani, Almas N. Memon, and Syed Z. Zaidi. 2013. Impact of GDP,
spending on R&D, number of universities and scientific journals on research publications among Asian
countries. PloS One, 8(6), e66449. doi:10.1371/journal.pone.0066449
Podlubny, I (2005) A note on comparison of scientific impact expressed by the number of citations in different
fields of science
http://arxiv.org/ftp/math/papers/0410/0410574.pdf
Schneider, J (2015) Publications or Citations – Does it matter? Beneficiaries in two different versions of a
national bibliometric performance model, an existing publication-based and a suggested citation-based model.
ISSI, Bogaziçi University Printhouse, (2015), from: http://www.issi2015.org/files/downloads/all-
papers/0477.pdf
Sivertsen, G. (2009). A Bibliometric Funding Model based on a National Research Information System.
Norwegian Institute for Studies in Innovation, Research and Education. Retrieved 23-6-0015, from:
http://www.issi2009.org/agendas/issiprogram/public/documents/ISSI%202009%20Sivertsen%20Vista-
094456.pdf
Strandskov, J. (2011). Referat af Institutledermøde den 6. december 2011, Syddansk Universitet. Retrieved 21-
04-2015, from: board.sam.sdu.dk/Download.mvc/2889
UFM (2015a). Basismidler efter kvalitet. Retrieved 22-6-2015, from: http://ufm.dk/uddannelse-og-
institutioner/videregaende-uddannelse/universiteter/okonomi/basismidler-efter-kvalitet
61
LINKS
The Ref indicator explained: http://www.ref.ac.uk/
Sherpa/Romeo. Publisher Copyright policies & self-archiving
http://www.sherpa.ac.uk/romeo/search.php
Higher Education Funding council for England:
http://www.hefce.ac.uk/news/newsarchive/2015/Name,103785,en.html
The BOF indicator explained (Dutch): http://data-
onderwijs.vlaanderen.be/edulex/document.aspx?docid=14492
The Norwegian Indicator: https://www.regjeringen.no/no/tema/helse-og-
omsorg/sykehus/nasjonalt-system-for-maling-av-forskning/id446980/#Ind
BFI: http://ufm.dk/forskning-og-innovation/statistik-og-analyser/den-
bibliometriske-forskningsindikator
62
Bibliometric indicators of quality?

More Related Content

What's hot

Scopus: Research Metrics and Indicators
Scopus: Research Metrics and Indicators Scopus: Research Metrics and Indicators
Scopus: Research Metrics and Indicators Michaela Kurschildgen
 
Abstract and i ndexing
Abstract and i ndexingAbstract and i ndexing
Abstract and i ndexingMohit Kumar
 
Journal Impact Factors and Citation Analysis
Journal Impact Factors and Citation AnalysisJournal Impact Factors and Citation Analysis
Journal Impact Factors and Citation Analysisrepayne
 
Metric Fields in Information Science
Metric Fields in Information ScienceMetric Fields in Information Science
Metric Fields in Information ScienceGladys Wakat
 
Seminar ppt hazards to library
Seminar ppt hazards to librarySeminar ppt hazards to library
Seminar ppt hazards to libraryNagendra N
 
Bibliometrics - an overview
Bibliometrics - an overviewBibliometrics - an overview
Bibliometrics - an overviewclaudia cavicchi
 
Scientometric Analysis
Scientometric AnalysisScientometric Analysis
Scientometric Analysissumitbanshal
 
Digital Library Initiatives in India : An Overview
Digital Library Initiatives in  India : An OverviewDigital Library Initiatives in  India : An Overview
Digital Library Initiatives in India : An OverviewManoj Kumar Sinha
 
FRSAD Functional Requirements for Subject Authority Data model
FRSAD Functional Requirements for Subject Authority Data modelFRSAD Functional Requirements for Subject Authority Data model
FRSAD Functional Requirements for Subject Authority Data modelMarcia Zeng
 
Bibliometric Tools
Bibliometric ToolsBibliometric Tools
Bibliometric ToolsUCT
 
What is bibliometrics and how does it work?
What is bibliometrics and how does it work?What is bibliometrics and how does it work?
What is bibliometrics and how does it work?Ciarán Quinn
 
National social science documentation centre (nassdoc )
National social science documentation centre (nassdoc )National social science documentation centre (nassdoc )
National social science documentation centre (nassdoc )GordonAmidu
 
Virtual reference srevices
Virtual reference srevicesVirtual reference srevices
Virtual reference srevicesiqra Mubeen
 

What's hot (20)

ISO 2709
ISO 2709ISO 2709
ISO 2709
 
bibliometrics
bibliometricsbibliometrics
bibliometrics
 
Unit 6
Unit 6Unit 6
Unit 6
 
Scopus: Research Metrics and Indicators
Scopus: Research Metrics and Indicators Scopus: Research Metrics and Indicators
Scopus: Research Metrics and Indicators
 
Abstract and i ndexing
Abstract and i ndexingAbstract and i ndexing
Abstract and i ndexing
 
Journal Impact Factors and Citation Analysis
Journal Impact Factors and Citation AnalysisJournal Impact Factors and Citation Analysis
Journal Impact Factors and Citation Analysis
 
Metric Fields in Information Science
Metric Fields in Information ScienceMetric Fields in Information Science
Metric Fields in Information Science
 
Bibliometric study
Bibliometric studyBibliometric study
Bibliometric study
 
Seminar ppt hazards to library
Seminar ppt hazards to librarySeminar ppt hazards to library
Seminar ppt hazards to library
 
BIBLIOMETRICS LAWS
BIBLIOMETRICS LAWSBIBLIOMETRICS LAWS
BIBLIOMETRICS LAWS
 
Bibliometrics - an overview
Bibliometrics - an overviewBibliometrics - an overview
Bibliometrics - an overview
 
Scientometric Analysis
Scientometric AnalysisScientometric Analysis
Scientometric Analysis
 
FRBR
FRBRFRBR
FRBR
 
Digital Library Initiatives in India : An Overview
Digital Library Initiatives in  India : An OverviewDigital Library Initiatives in  India : An Overview
Digital Library Initiatives in India : An Overview
 
FRSAD Functional Requirements for Subject Authority Data model
FRSAD Functional Requirements for Subject Authority Data modelFRSAD Functional Requirements for Subject Authority Data model
FRSAD Functional Requirements for Subject Authority Data model
 
Bibliometric Tools
Bibliometric ToolsBibliometric Tools
Bibliometric Tools
 
What is bibliometrics and how does it work?
What is bibliometrics and how does it work?What is bibliometrics and how does it work?
What is bibliometrics and how does it work?
 
Scopus
ScopusScopus
Scopus
 
National social science documentation centre (nassdoc )
National social science documentation centre (nassdoc )National social science documentation centre (nassdoc )
National social science documentation centre (nassdoc )
 
Virtual reference srevices
Virtual reference srevicesVirtual reference srevices
Virtual reference srevices
 

Viewers also liked

Bib & ethics wildgaard
Bib & ethics wildgaardBib & ethics wildgaard
Bib & ethics wildgaardLorna Wildgaard
 
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...Stefanie Haustein
 
Cost of quality concept and advantages
Cost of quality   concept and advantagesCost of quality   concept and advantages
Cost of quality concept and advantagesImran Jamil
 
07 09 04 Ctqi Standard
07 09 04 Ctqi Standard07 09 04 Ctqi Standard
07 09 04 Ctqi StandardKieran F Ring
 

Viewers also liked (6)

Bib & ethics wildgaard
Bib & ethics wildgaardBib & ethics wildgaard
Bib & ethics wildgaard
 
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmet...
 
Cost of quality concept and advantages
Cost of quality   concept and advantagesCost of quality   concept and advantages
Cost of quality concept and advantages
 
Cost of quality
Cost of qualityCost of quality
Cost of quality
 
Cost of quality
Cost of qualityCost of quality
Cost of quality
 
07 09 04 Ctqi Standard
07 09 04 Ctqi Standard07 09 04 Ctqi Standard
07 09 04 Ctqi Standard
 

Similar to Bibliometric indicators of quality?

Handbook on constructing composite indicators
Handbook on constructing composite indicatorsHandbook on constructing composite indicators
Handbook on constructing composite indicatorssagittarian
 
Oecd uni indcollaboration_ch1_website
Oecd uni indcollaboration_ch1_websiteOecd uni indcollaboration_ch1_website
Oecd uni indcollaboration_ch1_websiteslideshow19
 
Science &Technology policy in Italy: The role of evaluation
Science &Technology policy in Italy:  The role of evaluationScience &Technology policy in Italy:  The role of evaluation
Science &Technology policy in Italy: The role of evaluationGiuseppe De Nicolao
 
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...Liam Cleere University College Dublin’s Senior Manager for Research Analytics...
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...IrishHumanitiesAlliance
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxelishaoatway
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxsheronlewthwaite
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxalanrgibson41217
 
Uses and misuses of quantitative indicators of impact
Uses and misuses of quantitative indicators of impactUses and misuses of quantitative indicators of impact
Uses and misuses of quantitative indicators of impactBerenika Webster
 
Presentación SocietalImpact Daniel Torres
Presentación SocietalImpact Daniel Torres Presentación SocietalImpact Daniel Torres
Presentación SocietalImpact Daniel Torres EC3metrics Spin-Off
 
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise DataConstructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise Dataronicky
 
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...OpenAIRE
 
SIOR: recognition of the social impact of research through open access resources
SIOR: recognition of the social impact of research through open access resourcesSIOR: recognition of the social impact of research through open access resources
SIOR: recognition of the social impact of research through open access resourcesORCID, Inc
 
Doctoral Dissertation - Pablo Migliorini - July 2015
Doctoral Dissertation - Pablo Migliorini - July 2015Doctoral Dissertation - Pablo Migliorini - July 2015
Doctoral Dissertation - Pablo Migliorini - July 2015Pablo Migliorini
 
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and Impact
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and ImpactCecilia Tsui 2008 Higher Education Development: Quality, Relevance and Impact
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and ImpactCeciliaTsui42
 
Statistics for-effective-regional-policy
Statistics for-effective-regional-policyStatistics for-effective-regional-policy
Statistics for-effective-regional-policyOECD Governance
 
Altmetrics, Impact Analysis and Scholarly Communication
Altmetrics, Impact Analysis and Scholarly Communication �Altmetrics, Impact Analysis and Scholarly Communication �
Altmetrics, Impact Analysis and Scholarly Communication Michelle Willmers
 
Facing the future: Sense-making in Horizon Scanning
Facing the future: Sense-making in Horizon ScanningFacing the future: Sense-making in Horizon Scanning
Facing the future: Sense-making in Horizon ScanningTotti Könnölä
 

Similar to Bibliometric indicators of quality? (20)

ETF Manual on the Use of Indicators
ETF Manual on the Use of IndicatorsETF Manual on the Use of Indicators
ETF Manual on the Use of Indicators
 
Handbook on constructing composite indicators
Handbook on constructing composite indicatorsHandbook on constructing composite indicators
Handbook on constructing composite indicators
 
Oecd uni indcollaboration_ch1_website
Oecd uni indcollaboration_ch1_websiteOecd uni indcollaboration_ch1_website
Oecd uni indcollaboration_ch1_website
 
Science &Technology policy in Italy: The role of evaluation
Science &Technology policy in Italy:  The role of evaluationScience &Technology policy in Italy:  The role of evaluation
Science &Technology policy in Italy: The role of evaluation
 
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...Liam Cleere University College Dublin’s Senior Manager for Research Analytics...
Liam Cleere University College Dublin’s Senior Manager for Research Analytics...
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
 
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docxIEP Team Members 10.0 Summary comprehensively and correctly id.docx
IEP Team Members 10.0 Summary comprehensively and correctly id.docx
 
Uses and misuses of quantitative indicators of impact
Uses and misuses of quantitative indicators of impactUses and misuses of quantitative indicators of impact
Uses and misuses of quantitative indicators of impact
 
Societal Impact
Societal Impact Societal Impact
Societal Impact
 
Presentación SocietalImpact Daniel Torres
Presentación SocietalImpact Daniel Torres Presentación SocietalImpact Daniel Torres
Presentación SocietalImpact Daniel Torres
 
RTP 18-19 - The Research Excellence Framework (REF) 2021
RTP 18-19 - The Research Excellence Framework (REF) 2021RTP 18-19 - The Research Excellence Framework (REF) 2021
RTP 18-19 - The Research Excellence Framework (REF) 2021
 
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise DataConstructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
 
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...
20190527_Marc Vanholsbeeck_Open Science monitoring and the notion of research...
 
SIOR: recognition of the social impact of research through open access resources
SIOR: recognition of the social impact of research through open access resourcesSIOR: recognition of the social impact of research through open access resources
SIOR: recognition of the social impact of research through open access resources
 
Doctoral Dissertation - Pablo Migliorini - July 2015
Doctoral Dissertation - Pablo Migliorini - July 2015Doctoral Dissertation - Pablo Migliorini - July 2015
Doctoral Dissertation - Pablo Migliorini - July 2015
 
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and Impact
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and ImpactCecilia Tsui 2008 Higher Education Development: Quality, Relevance and Impact
Cecilia Tsui 2008 Higher Education Development: Quality, Relevance and Impact
 
Statistics for-effective-regional-policy
Statistics for-effective-regional-policyStatistics for-effective-regional-policy
Statistics for-effective-regional-policy
 
Altmetrics, Impact Analysis and Scholarly Communication
Altmetrics, Impact Analysis and Scholarly Communication �Altmetrics, Impact Analysis and Scholarly Communication �
Altmetrics, Impact Analysis and Scholarly Communication
 
Facing the future: Sense-making in Horizon Scanning
Facing the future: Sense-making in Horizon ScanningFacing the future: Sense-making in Horizon Scanning
Facing the future: Sense-making in Horizon Scanning
 

Bibliometric indicators of quality?

  • 2. OUTLINE INDICATORS OF QUALITY?? • Methods of quantitative evaluation • University level – Macro bibliometric analysis • Individual level – Micro bibliometric analysis Lorna Wildgaard Royal School of Library and Information Science Lorna.wildgaard@hum.ku.dk Kildekritik, F2016
  • 3. STUDY OF QUANTITATIVE ASPECTS OF: informetrics bibliometrics scientometrics webometrics cybermetrics © Björneborn altmetrics Information in any form by any social group Production, dissemination & use of recorded information Science as a discipline or economic activity Politico-economical aspects. Construction & use of information resources & tech. on the internet Web documents, text & multi-media, blogs, tags, wikis.. Dissemination, use & online activity of scholarly content
  • 4. SO METRICS ARE: ”The application of mathematical and statistical methods to books and other media of communication.” Pritchard, A (1969), Journal of Documentation, 25(4):348-9
  • 6. PERCENT DISCIPLINARY COLLABORATION C= Nm Nm+Ns Nm = Number of multi-authored papers in the discipline, eg. 22929 Ns=Number of single authored papers in the discipline, eg. 9325 C= 22929 22929+9325 = 0.71 * 100 = 71%
  • 7. S c (i) = γ ∗ (Y (now) − Y (i) + 1)−δ ∗ |C(i)| Weighted citation score of an article Actual Year minus publication year Citation count Coefficient set at 4, i.e. for an article published during the current year, its citations account four times
  • 8. WHAT IS A PUBLICATION? A scholarly publication is a an expression of the state of the researcher at a particular time (Price 1970)….the product of scientific research (Cawkell 1976; Lazarev 1996)….what consitutes a publication is different in the scientific disciplines (Castellini 2014; Hicks 2012; Tinkler 2011; Frandsen & Nicolaisen 2011)….different publication types indicate different methodologies (Grant & Booth 2009;Dzombak 2013)….Publications should be conceptualized as a discrete set of objects so they can be counted in aggregate (Skupin 2009)….output in a citation index (Antonakis & Lalive (2008). What constitutes a ”publication” should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012)
  • 9. WHAT IS A CITATION? Citations reflect som sort of cognitive influence (Moed 2005)….the strength of the influence is not known (Martyn 1964)….income or reward (Ravetz 1971)….meaning of citation is governed by paradigmatic & social norms (MacRoberts & MacRoberts 1996; Cronin 2000)….acknowledgement of intellecual debt to other works (Merton 1971)…. Markers or symbols of influence in the aggregate (Small 1978;1987)….citation impact (Martin & Irvine 1983)….rhetorical devices (Luukkonen 1997; Latour 1987)….authority (Gilbert) quality sensor mechanisms (Lindsey 1978)….A manipulable measure of something or other (Singleton 1976)….influence on average (Nicolaisen 2004)…citations are arbitrary (Leimu & Korcheva 2005)… markers (footprints) of scientific communication (Garfield 1979)….income (Albarrán 2011)…… The concept of citation is open for interpretation, of which there are numerous, and it is impossible to exclusively link citations to quality
  • 10. WHAT IS A RESEARCHER? Dias 10 21/02/2016
  • 11. WARM UP EXERCISE https://todaysmeet.com/Warm_Up Handout: Researcher evaluation 1. Suggest indicators that could measure attributes, skills, interests and achievements. 2. Consider how to indicate the quality of these attributes, skills, interests & achievements.
  • 12. EVALUATION OUTPUT DATA MODEL MOTIVATION TO PUBLISH MOTIVATION TO CITE INTERPRETATION OF ALI LEGITIMATE LINK TO REAL WORLD PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS EXOGENOUS VARIABLES WHAT IS MISSING INDEXING POLICY THE PROBLEM
  • 13. COFFEE BREAK: Meet back here in 10 minutes
  • 15. RESEARCH PERFORMANCE EVALUATION SYSTEMS -Register the amount & type of a scholar’s publications. REF (UK): Panel-based evaluation every 6th year. Quality=peer review. BFO (Flanders): Citation-based Quality=reception & use ERA (Aus)=Publication-based (udifferentiated). The number of publications dictates the amount of funds given to the university & the assessment of a researcher’s productivity. Quality= not assessed
  • 16. Norwegian indicator 2004 (differentiated) Tiered system of publication types and sources. Aim: increase publications in international journals and establish a registration system in which institutes can validate & verify publications as well as see other institutions publications Quality=not addressed BACKGROUND FOR BFI BFI (DK) 2009-2012 (differentiated) Tiered system of publication types and sources Aim: Distribute funding based on the quality of research results (UFM 2015a). Universities are judged based on published research. Quality= explicit assumption: the journal's status is assumed to say something about the individual publication’s quality (UFM 2015a). 16
  • 17. BFI Norge Quality indicator Publications indicator 25% of a sub-pool of basic funds, amount changes each year 2% of all university funds each year Domain neutral Normalised. Benchmarked against total publication points and domain share Unclear, complicated Clear, simple Publication performance Stimulate publication in international resources Accountability model, aimed at distributing funds, production measured in Danish Crowns and legitimacy Registration model Fractionalised count ”Whole” counting International/eksternal collaboration rewarded with more points Points normalised to the domain w.r.t. speed of publication, collaboration traditions and publication type Book publishers on level 1 & 2 Book publishers on Level 1
  • 18. TEACHING 40% FACULTY WORK 23% ADMIN. 18% RESEARCH & PUBLICATION 12% NETWORK 7% WHAT IS THE BFI MEASURING? HOW RESEARCHERS SPEND THEIR TIME: WEEKLY AVERAGE Ziker (2014) https://thebluereview.org/faculty-time-allocation/18 BFI
  • 19. GROUP AREA FIELD COVERAGE 1-19 HUM Language, Culture, History, Linguistics & Semiotics, Literature, Dance, Music, Theater, Art History And Art, Cultural Studies, Media & Communication, Film, Rhetoric, Theology, Philosophy, Archeology, Pedagogy, Education, Anthropology, IT & Humanities 20-27 SOC. SCI Law, Sociology, Social Work, Political Science, Geography, Social Economics, Psychology, Library & Information Science 28-46 SCI/TEK Geo-physics, Physics, Astronomy, Math, Chemistry, Biology, Environmental Science, Agriculture, Molecular Science, Pharmacology, Civil Engineering, Transport, Electronics, Energy, Chemical Engineering, Nanoscience, Medical Technology, System Development And Entrepreneurship, Food Science 47-62 SUND Medicine, Surgery, Anesthesiology, Neurology, Ear-nose-throat, Eye Disease, Gynecology, Dermatology, Pediatrics, Pharmacology, Dentistry, Nursing, Public Health, Veterinary, Pathology, General Medicine 63-68 OTHER Gender Studies, Architecture, Town Planning, Sport, Digital Communication, Information Systems & IT Management, Science Studies And Research Analysis 19
  • 20. BFI STRUCTURE: PUBLICATION TYPES Publication types N. 1 N.2 Monography in ISBN title (inc. reports, ex. Text books) 5 8 Monography in book series 5 8 Contribution to anthology, ISBN title (inc. introductory & final chapters; book chapters; ex., abstracts, lecture manuscripts, PPTs, encyclopedia, preface/afterword, commentaries, editorial work) 0.5 2 Contribution to anthology, bookseries 1 3 Contribution to a conference, series 1 3 Contribution to a conference 0.5 2 Article (inc. letter, review ex., abstracts, editorial letter, commentary, book review) 1 3 Doctorate 5 Patent 1 Publication types registered in PURE, PURE4 42 basic publishing types: http://bibliotek.science.ku.dk/forskdok/docs/metadatamodel_P4_LIFE_GR_270312.pdf20
  • 21. CALCULATION OF BFI POINTS Points are shared between universities dependent on how many authors belonging to the university are listed on the publication: A book on level 2 gives 8 points. There are 3 authors from AU, 2 from CPU & 1 from a foreign university. AU = 3/6 points CPU=2/6 points 8/6 = 1.3 points to each author Collaboration with authors from other universities, Danish or foreign, is rewarded, The total number of points are multiplied by 1.25. AU total points= 1.3 * 3 =3.9 * 1.25 = 4.87 points CPU total points = 1.3 * 2 =2.6 * 1.25 = 3.25 points BFI is domain neutral (funds are shared within domain and not across domains) 21
  • 22. DISTRIBUTION KEY (fordelingsnøglen) 2009 45% STÅ INCOME 20% EKSTERNAL 10% PhD 25% BFI 2015 2008 50% STÅ INCOME 40% EKSTERNAL 10% PhD 2000 Tax catalogue: http://ufm.dk/uddannelse-og-institutioner/videregaende- uddannelse/universiteter/okonomi/uddannelsesbevillinger 22
  • 23. WHAT HAPPENS AFTER 2015? BFI must only distribute new funds and not alter the existing taxometer for distribution of basic university funds (DU 2009) • New funds = globalization pool • The globalization pool is expected to be emptied in 2015 • Suggested 25% BFI distributions parametre is supplemented by taking funds from external grants • Basis funding is grouped together with other funding and therefore effects distribution and share. fx: HUM greatest operational costs, greatest number of students, lowest STÅ tax and lowest amount of funds invested in researcher education, researcher support, administration and traditionally the lowest number of publications. http://www.altinget.dk/forskning/artikel/basismidler-fylder- mindre-paa-universiteterne 23
  • 24. COLLABORATION PARAMETER Publications Level Total Points % National points Collaborative publications % Collaborative publications 1 2 NAT/TEK 2921 1596 1325 2973.99 24 2406 82 SUND 3437 2379 1058 2851.26 42 3056 88 HUM 963 625 338 1502.72 34 185 19 SAM 739 448 291 1141.47 21 225 30 I ALT 8060 5048 3012 8469.44 30% 5872 72% *Universities share of the total number of national points, given in percent https://bfi.fi.dk/Publication/NationalAnalysis 24
  • 25. PUBLICATION POINTS (UFM 2015) % diff. i p.p 2013-2014 % diff. antal p. 2013-2014 Aalborg U +35.3 +37 Aarhus U +4.1 +2.6 CBS +20.2 +11.6 DTU -0.3 +3.7 ITU +19.5 +12.5 KU +0.3 -3.7 RUC +7.7 -4.9 SDU +16.1 -16.4 DOMAIN % diff. 2013-2014 NAT/TEK -0.5 SUND +16.9 HUM +15.3 SAMF +12.5 25
  • 26. EVALUATION OF BFI Good focus on strategic publishing. BFI works as an incitement to publish in certain resources. Yet, the development of longterm, dialogue based communication forms with public society could be ignored in a quest to fulfill the requirement of selling science through an unidirectional intemediatory (red.: journals). (Schneider, 2015) 26
  • 27. VISIBILITY OF HUM IN CITATION INDICES 0 50 100 150 200 250 300 350 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Numberofpublications Publication year HUM i WOS HUM i Scopus BFI P=963 BFI P=813 BFI 27
  • 28. EVALUATION OF BFI UFM(2015) 1. BFI lacks transparancy 2. Need for better communication between stakeholders 3. Citation indicators could be advantages in some domains. This will enable analysis of international competition. The performance of BFI in the model to distribute funds was not assessed 28
  • 29. CONSIDERATIONS BFI results in an increase of research publications in level 1 and 2 sources, and increased collaboration (Ingwersen & Larsen, 2014) BFI results in an increased attention on the researcher employment promotion funding applications Under contract to publish in BFI resources and produce publication points(AAU Bestyrelse, 2014; FOU., 2013; Strandskov, 2011) 29
  • 30. LOOKING TO THE FUTURE… Does BFI prioritize what is measurable over what should be measured but is difficult to quanitify? (Aagaard 2012) • Leiden manifesto: http://www.leidenmanifesto.org/ • Snowball Metrics: http://www.snowballmetrics.com/ • Responsible Metrics https://responsiblemetrics.org/
  • 31. COFFEE BREAK: Meet back here in 10 minutes
  • 33. TYPES OF INDICATORS Hybrid metrics PUBLICATION METRICS count production, collaboration, & contribution CITATION METRICS count citations, self- citations, adjust for collaboration Table of indicators: http://tinyurl.com/nj4mvca Ranking, growth, proxy for excellence, effect, field comparisons & quality
  • 34. INTERNAL CHARACTERISTICS OF ALI Wildgaard et al (2014) A review of the characteristics of 108 author- level indicators. Scientometrics, 101(1):125-158
  • 35. Age: years since first publication (WoS) = 2016-2009 = 7 Papers: all publications (WoS)=13 Citations: total citations (WoS)= 190 1st subject area=Anesthesiology 2nd subject area=Neuroscience 3rd subject area=Surgery INDEX OF QUALITY AND PRODUCTIVITY http://tinyurl.com/zlxzyz9 Antonakis, J. and Lalive, R. (2008), Quantifying Scholarly Impact: IQp Versus the Hirsch h. J. Am. Soc. Inf. Sci., 59: 956–969. doi: 10.1002/asi.20802 http://www.hec.unil.ch/jantonakis/IQp%20calculator%20version %202008.xls
  • 36. INDEX OF QUALITY AND PRODUCTIVITY 190 Cited 1.44 times more than average paper in the specialty Number of papers performing above average for the specialty JIF of 3 subject categories
  • 37. ANOTHER ASPECT OF QUALITY? WEB/ALTMETRICS
  • 38. WEB 2.0 = PARTICIPATIVE WEB ”… supports, extends, or derives added value from human social behaviour …”
  • 39. TWO ELEMENTS IN NETWORKS Node Nodes represent things that relate somehow to one another Tie Tie represents relations between nodes © James Cook https://www.youtube.com/channel/UCerAw4EfTOnYYxLLPZAzMxQ
  • 40. TWO KINDS OF NETWORKS DIRECTED NETWORKS Networks in which the tie has a driection (digraphs), eg. Linking to a webpage, downloading a document, citation networks UNDIRECTED NETWORKS Networks in which the tie has no direction eg. friends on Facebook, © James Cook
  • 41. INDUCED HOMOPHILY Tendency for ties to form similar others because similar others present the social environments (group, community, society) No wonder blues are mostly tied to blues…there are hardly any reds out there! © James Cook
  • 42. CHOICE HOMOPHILY A tendency to choose to form ties with similar others even when different others are available in the social environment (group, community, society) © James Cook
  • 43. PRINCIPLES OF NETWORKS • Ties should be straight • Ties should not be far from each other • Ties should not cross or touch • Ties should be easy to follow from node to node • Nodes that connect should be close • Similar nodes should be close to one another • Central nodes should be in the center © James Cook
  • 46. Quick-Diverse-Broad A record of attention: how many people have been exposed to and engaged with a scholarly output in the news, blogs, and on Twitter; article pageviews and downloads; GitHub repository watchers. A measure of dissemination: where and why a piece of research is being discussed and shared, both among other scholars and in the public sphere. An indicator of influence and impact: Some altmetrics can signal that research is changing a field of study, the public’s health, or having any other number of tangible effects upon larger society, eg. references in public policy documents; or commentary from experts and practitioners.
  • 47. An output • Journal Article • Dataset • Poster An identifier attached to the output • DOI • PubMedID Mention in a source • Social media • Blogs • Wikipedia • Media API stands for Application Programming Interface
  • 48. ALTMETRIC.COM The colours of the donut each represent a different source of information. The attention score is a weighted count of all of the attention a research output has received. http://tinyurl.com/z7zozr5
  • 49. Demographic breakdown Count % Members of the public 153 18 Scientists 31 16 Science communicators (bloggers, journalists, editors) 5 3 Practitioners (doctors, other healthcare professionals) 3 2
  • 50. Indicators are designed to measure particular aspects of the effect of a researcher’s work – over time, to field, as quality, ranking all or selected works, co-authorship etc. Judgements based on indicators can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual. ALI have to be methodologically sound WHAT WE HAVE TO BE AWARE OF
  • 51. 51 ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF- WORTH ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS DOCUMENTING BEING OUT- PERFORMED IS DETRIMENTAL TO A RESEARCHER’S SELF-DEFINITION ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION SUCCESS IS DEFINED AS WELL IN THE SYSTEM EXTERNAL CHARACTERISTICS OF ALI
  • 52. 1. Transparency Know which data is used to compute the indicator Understand the math and the inferences 2. Demographics Be aware if demographics affect the indicator 3. Motive Indicators must fit the objectives of the evaluation 4. Diversity Choose indicators that fit the discipline/publication 5. Openess Make the limitations of indicator explicit, use supplementary indicators APPROPRIATE INDICATORS
  • 53. “CAN WE ASSESS THE BEAUTY OF THE MONA LISA BY COUNTING THE NUMBER OF VISITORS TO THE LOUVRE?” GROUP DISCUSSION. WRITE YOUR IDEAS AT https://todaysmeet.com/Indicators_of_Quality
  • 59. REFERENCES Aagaard, K & Mejlgaard, N (2012) Dansk forskningspolitik efter årstusinskiftet. Aarhus Universitetsforlag AAU (2013). Newsletter from the Rectorate, October 2013. Retrieved 23-6-2015 from: http://www.intern.aau.dk/ansatte/ledelsen-AAU/aau- rektoratet/rektoratets-nyhedsbrev/vis-nyhedsbrev//nyt-fra-rektoratet-oktober- 2013---newsletter-from-the-rectorate-october-2013.cid97315#bfi Barbulescu, Roxana. 2011. European Research Council’s Grant for Advanced Investigators in Social and Political Sciences. European University Institute. DS 2014: Svagtfald I statens forskningsbudget (Nyt fra Danmarks statistik, 28 jan 2014) http://www.dst.dk/pukora/epub/Nyt/2014/NR039_1.pdf DU. (2009). Politikpapir om model for fordeling af basismidler. Faggruppe 68. Retrieved 23-6-2015, from: http://faggruppe68.pbworks.com/f/Politikpapir_170409__til_hjemmesiden_.pdf59
  • 60. REFERENCES DU. (2012). Overblik over finanslovforslaget for 2013. Retrieved 23-6-2015, from: http://www.dkuni.dk/Politik/~/media/Files/Politiknotater/notat%20om%20ffl13%20(3 ).ashx Emmeche, C. (2014). Den Bibliometriske Forskningsindikator - fordele og ulemper. Faggruppe68. Retrieved 23-6-2015, from: http://faggruppe68.pbworks.com/w/page/6015700/Den%20Bibliometriske%20Forskn ingsindikator%20-%20fordele%20og%20ulemper FOU. (2013). Publiceringsstrategi for Rosklide Universitet 2015-2018. RUC Hearing. Retrieved 23-6-2015, from: www.ruc.dk/?eID=minute&m=1948&t=179&docID=30986 Hansen, L. (2009) ‖Hvorfor er en humanistisk artikel mere værd end en sundhedsvidenskabelig?‖ Indlæg på bloggen ‘Forskningsfrihed‘ 12/10-2009. Ingwersen & Larsen (2014) Influence of a performance indicator on Danish research production and citation impact 2000–12. Scientometrics, Online 19/4 2014; DOI 10.1007/s11192-014-1291-x. 60
  • 61. REFERENCES Meo, Sultan A., Abeer A. Al Masri, Adnan M. Usmani, Almas N. Memon, and Syed Z. Zaidi. 2013. Impact of GDP, spending on R&D, number of universities and scientific journals on research publications among Asian countries. PloS One, 8(6), e66449. doi:10.1371/journal.pone.0066449 Podlubny, I (2005) A note on comparison of scientific impact expressed by the number of citations in different fields of science http://arxiv.org/ftp/math/papers/0410/0410574.pdf Schneider, J (2015) Publications or Citations – Does it matter? Beneficiaries in two different versions of a national bibliometric performance model, an existing publication-based and a suggested citation-based model. ISSI, Bogaziçi University Printhouse, (2015), from: http://www.issi2015.org/files/downloads/all- papers/0477.pdf Sivertsen, G. (2009). A Bibliometric Funding Model based on a National Research Information System. Norwegian Institute for Studies in Innovation, Research and Education. Retrieved 23-6-0015, from: http://www.issi2009.org/agendas/issiprogram/public/documents/ISSI%202009%20Sivertsen%20Vista- 094456.pdf Strandskov, J. (2011). Referat af Institutledermøde den 6. december 2011, Syddansk Universitet. Retrieved 21- 04-2015, from: board.sam.sdu.dk/Download.mvc/2889 UFM (2015a). Basismidler efter kvalitet. Retrieved 22-6-2015, from: http://ufm.dk/uddannelse-og- institutioner/videregaende-uddannelse/universiteter/okonomi/basismidler-efter-kvalitet 61
  • 62. LINKS The Ref indicator explained: http://www.ref.ac.uk/ Sherpa/Romeo. Publisher Copyright policies & self-archiving http://www.sherpa.ac.uk/romeo/search.php Higher Education Funding council for England: http://www.hefce.ac.uk/news/newsarchive/2015/Name,103785,en.html The BOF indicator explained (Dutch): http://data- onderwijs.vlaanderen.be/edulex/document.aspx?docid=14492 The Norwegian Indicator: https://www.regjeringen.no/no/tema/helse-og- omsorg/sykehus/nasjonalt-system-for-maling-av-forskning/id446980/#Ind BFI: http://ufm.dk/forskning-og-innovation/statistik-og-analyser/den- bibliometriske-forskningsindikator 62