Keynote at NEFUS (Danish Network for Research Support services) Workshop on ethics and leadership in the use of bibliometric data. My talk discusses the challenges we face in bibliometric analysis of the individual researcher. The slides are a mixture of English and Danish. The workshop link is here: https://www.dfdf.dk/index.php/arrangementer/details/41-NEFUS?xref=35

2. BLASPHEMY
“the biases and deficiencies of individual citers are
repaired to a tolerable degree by the combined
activity of the many” (White, 2001), where
deficiencies are reduced to “random noise” (Cawkell,
1976) and “…references can be used on the
aggregate as an indicator of influence” (Small, 1987)

3. Introduktion af h-indeks i 2005, førte til en eksplosion
af FNI, der hævder at være mere robust, gyldig og
sofistikeret end de øvrige.
FNI er ødelæggende, tilskynder til en ‘gaming’
mentalitet og støtter universiteterne i at presse
medarbejdere til målrettet at øge deres indeks
og tilskynder forskningspolitik der overvåger
forskningsresultater på det individuelle niveau
(Dahler-Larsen, 2012; Collini, 2012).

4. Improvements on h
hw (weights citations); hn (field comparisons); ht (id
priority articles); hpd (seniority, 10 years); hc (seniority, 4
years); Q2 (number/impact papers in h); hα (granular h
ranks) hT (alternative h calculation)
Complimentary to h
m-quotient (h normalized for age); m (median C in h
core); A (average C in h core); R (square root of A); E
(effect papers not in h); AR (Citation intensity & age of
C); b (self citations, top papers); Rational h (distance to
higher h); Hm (multi-authorship); n (field comparison); h
sequences & matrices (field comparison); hf (field
comparison); Alternative h (multi authorship); Pure h
(multi authorship); Adapted pure-h (multi-authorship);
Dynamic h (compare peers); hmx (database comparison)

5. Narrative
IQP (average quality); Index Age & Productivity
(academic age on productivity & impact); %HCP (top
20%); Classification of durability (document type & field
comparison); DCI (citation age)
Replace h
g, gα (more granular than h); mg-quotient, AWCR
(normalized for age); rational g (distance to higher g); hg
(compare h & g values); h2 (cummulative acheivement);
ħ, x (compare field/seniority); ct, at(aging rate);
wu(excellent papers); π (production & impact); POP h,
AWCRpa (multi-authorship); AW (adjust highly cited
papers)

6. With no advisory boards, common standards or
contextual assessments, indicators are mostly
incomparable, which in fact impedes the development
of the field and makes the users of scientometric
results mistrustful.
(Vinkler, 1996).
Standarder vedrørende de etiske aspekter af
evaluerende bibliometri er igen nyligt blevet
foreslået.
(Bornmann, 2008; Bach, 2011; Furner, 2014; Hicks et
al, 2015)

7. Contextual bibliometrics
1994/1996
Matematiske standarder
for analyse,
præsentation og
fortolkning af data
2008
Etiske standarder til
evaluering af individer
2011
Begrebsramme om etik
& bibliometri ift.
fordeling af midler
2014
DORA: brug ikke JIF i
evaluering af individer
2012
Leiden Manifesto, Metric
Tide informeret brug og
formidling
2015
Contextual
bibliometrics
2016

9. Evalueringer baseret på indikatorer, kan føre til
antagelser om en forskers produktivitet og impact,
som kan være udokumenterede, og kan påvirke
forskerens selvopfattelse.

10. 10
FNI GIVER ET SNAP-SHOT AF
INDIVIDETS IMAGE OG CENTRALE
PERSONLIGHEDSTRÆK
SAMMENLIGNINGER KAN
EKSPONERE DEN ENKELTE
ALLE FORMER FOR DATA BRUGES TIL
AT ØGE FNI, INDIVIDETS VALIDITET
INDEN FOR DOMÆNET OG DERES
SELVVÆRD
FNI BERIGER EN EVALULERING MED
OBJEKTIVITET, REDUCERER KØN,
KULTURELLE OG RACEFORDOMME
DOKUMENTERE AT MAN IKKE
KLARER SIG BEDRE END ENS
KOLLEGAER KAN SKADE
FORSKERENS SELVOPFATTELSE
FNI BIDRAGER IKKE ALTID MED
VALUE-ADDED INFORMATION;
INFORMATION KAN VÆRE
REDUNDANT
SUCCES DEFINERES INDEN FOR
EVALUERINGSSYSTEMET

11. EVALUERING
OUTPUT
DATA
MODEL
FORTOLKNING
AFSPEJLER
REALITETER
SUCCES I SYSTEM vs
SUCCES I FELTET
FEJL/MANGLER
INDEKSERING
UDFORDRINGER

12. 24-10-2016 12
UDFORDRING #1: MODELLERNE
Forstå mekanismer i konstruktionen af
bibliometriske indikatorer.
Forstå hvad det er, vores data kan vise.

13. e=18
h=8
m=1
𝑿=3
CPP=18
t=4

14. S c (i) = γ ∗ (Y (now) − Y (i) + 1) − δ ∗ |C(i)|
Vægtet
citations score
for en artikel
Evaluerings år
minus
publikations år
Antal citationer
Valgbar koefficient (sat til 4), som gør at en
atikel fra evalueringsåret tildeles en faktor 4

15. 24-10-2016 15
UDFORDRING #2: DATA
”In time-keeping, in trading, in fighting, men
counted numbers; and finally, as the habit grew,
only numbers counted.”
Mumford, (2010)

16. 24/10/2016 16
Rå publikations/citationstal,
fortæller os ingenting om
forskningskvaliteten.
Observationer – studerer
processer og interaktioner.
Spørgeskemaer – identificerer
meninger, erkendelser og
forståelse.
Interviews – inviterer til feedback
som man normalt ikke vil få.

17. 24-10-2016 17
UDFORDRING #3: INDEKSERING
Indikatorer er i sig selv ikke neutrale!
Den største andel af indikatorer og databaser har
deres oprindelse i de nordlige lande, og afspejler
derfor karakteristika for videnskab i disse
regioner.

18. 24/10/2016 18
Hvordan kan vi udvikle
lokale systemer for at
sammenligne, skabe &
gen-sammenligne gen-
skabe indikatorværdier i
multifacetterede data, så vi
undgår ”én-dimensionelle
profiler”?

19. 24-10-2016 19
UDFORDRING #4: OUTPUT
Det vi ikke tæller, er usynligt.

20. ½
24/10/2016 20
Hvad er en publikation?
Hvad er en citation?
Hvad er en forsker?

21. 24-10-2016 21
UDFORDRING #5: EVALUERING
“The assessment itself is completely artificial. It’s not
ranking researchers in accordance with their ability to
develop, reach their potential, and explore their creative
interests. Those things you’re not testing..... it’s a rank
that’s mostly meaningless. And the very ranking itself is
harmful. It’s turning us into individuals who devote our
lives to achieving a rank. Not into doing things that are
valuable and important.”
Noam Chomsky (2015)

22. 24-10-2016 22
Mød dem der bliver evalueret.
Undersøg hvorfor de publicerer der hvor de gør.
Argumenter for relevans af indikatorer.
Rådgiv.
Kommuniker/formidl.

23. CHOOSE WISELY
24-10-2016 23
Transparens
Demografi
Motivation
Mangfoldighed
Åbenhed

24. Multi-dimensional research assessment matrix
(Moed, 2011)
UNIT OF
ASSESSMENT
PURPOSE OUTPUT BIBLIOMETRIC
INDICATOR
OTHER
INDICATORS
INDIVIDUAL
ALLOCATE
RESOURCES
RESEARCH
PRODUCTIVITY
PUBLICATIONS PEER REVIEW
RESEARCH
GROUPS
IMPROVE
PEFORMANCE
QUALITY,
SCHOLARLY
IMPACT
JOURNAL
CITATION
IMPACT
PATENTS,
LICENCES, SPIN
OFFS
DEPARTMENT
INCREASE
MULTI-DISCIPL.
RESEARCH
INNOVATION
& SOCIETAL
BENEFIT
ACTUAL
CITATION
IMPACT
INVITATIONS
FOR
CONFERENCES
INSTITUTION
INCREASE
REGIONAL
ENGAGEMENT
SUSTAINABIL-
ITY & SCALE
INT. CO-
AUTHORSHIP
EXTERNAL
RESEARCH
INCOME
RESEARCH
FIELD
PROMOTION,
HIRING
RESEARCH
INFRASTRUCT
CITATION
”PRESTIGE”
PHD
COMPLETION
RATES

25. Hvordan kan tilgængelighed
af kontekstuel information
forbedres?
Hvordan kan vi vejlede
individer i brugen af deres
information?
Hvordan opfylder vi løftet om
informeret peer review?
Hvordan kan vi influere
institutioners tilgang til
forskningsevaluering?

26. Contextualized bibliometrics
24-10-2016 26
Levér information der kan
eksploreres.
Undgå for meget vægt på det der let
kan kvalificeres/tælles.
Vælg simple indikatorer frem
for komplicerede indikatorer.
Vær kritisk. Vær proaktiv.
CALL FOR ACTION

27. 24/10/2016 27
Som bibliometrikere, skal vi forpligte os til
at underbygge meningsfulde sandheder.
Lorna Wildgaard, Ph.D
Lorna.Wildgaard@hum.ku.dk

28. REFERENCER
• Bach, J. F. (2011). On the proper use of bibliometrics to evaluate
individual researchers. Académie des sciences. Retrieved 23-6-
2015 from:
http://www.academie-
sciences.fr/activite/rapport/avis170111gb.pdf
• Bornmann, L., Mutz, R., Neuhaus, C., and Daniel, H-D. (2008b).
Citation counts for research evaluation: standards of good
practice for analyzing bibliometric data and presenting and
interpreting results. Ethics in Science and Environmental Politics,
8(1), 93-102.
• Cawkell, A. E. (1976). Understanding science by analysing its
literature. The Information Scientist, 10(1), 3-10.
24/10/2016 28

29. REFERENCER
• Chomsky, N. (2015) Creative by Nature, Blog post:
https://creativesystemsthinking.wordpress.com/2015/02/21/noa
m-chomsky-on-the-dangers-of-standardized-testing/
• Collini, S. (2012). Bibliometry. In What are universities for?
(pp.120-131) London: Penguin.
• Dahler-Larsen, P. (2012). The Evaluation Society. California:
Stanford University Press.
• Furner, J. (2014). The Ethics of Evaluative Bibliometrics. In
B.Cronin & C. Sugimoto (Eds.), Beyond Bibliometrics: Harnessing
Multidimensional Indicators of Scholarly Impact (pp. 85-107).
Massachusetts: MIT Press.
• Hicks, D. Wouters, P. Waltman, Ludo. de Rijcke, S., and Rafols, I.
(2015). Bibliometrics: The Leiden Manifesto for research metrics.
Nature, 520(7548), 429-431.
24/10/2016 29

30. REFERENCER
• Moed, H. (2011). The multi-dimensional research assessment
matrix. Research Trends, 23 (May 2011):
http://www.researchtrends.com/issue23-may-2011/the-multi-
dimensional-research-assessment-matrix/
• Mumford, L. (ed.2010). Technics and Civilization. University of
Chicago Press. p.22
• White, H. D. (2001). Authors as citers over time. Journal of the
American Society for Information Science and Technology, 52(2),
87-108.
• Small, H. G. (1987). The significance of bibliographic references.
Scientometrics, 12(5-6), 339-341.
• Vinkler, P. (1996). Some practical aspects of the standardization
of scientometric indicators. Scientometrics, 35(2), 235-245.
24/10/2016 30