Haustein, S. (2015). Scientific Interactions and Research Evaluation: From Bibliometrics to Altmetrics
Keynote at ISI2015 in Zadar, Croatia
http://isi2015.de/?session=keynote-c-i
Abstract. Since its creation 350 years ago, the scientific peer-reviewed journal has become the central and most important form of scholarly communication in the natural sciences and medicine. Although the digital revolution has facilitated and accelerated the publishing process by moving from print to online, it has not changed the scientific journal and scholarly communication as such. Today publications and citations in peer-reviewed journals are considered as indicators of scientific productivity and impact and used and misused in research evaluation. As scholarly communication is becoming more open and diverse and manuscripts, data, presentations and code are shared online, the altmetrics and open science movement demand the adaption of evaluation practices. Parallels are drawn between the early days of bibliometrics and current altmetrics research highlighting possibilities and limitations of various metrics and warning against adverse effects.
Top profile Call Girls In Bihar Sharif [ 7014168258 ] Call Me For Genuine Mod...
Scientific Interactions and Research Evaluation: From Bibliometrics to Altmetrics - Keynote ISI2015
1. Scientific interactions
and research evaluation:
from bibliometrics to altmetrics
Stefanie Haustein
stefanie.haustein@umontreal.ca
@stefhaustein
crc.ebsi.umontreal.ca/sloan
4. Invisible Colleges
Père Marin Mersenne
(1588-1648)
Henry Oldenburg
(1619-1677)
http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg
http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
5. Invisible Colleges
Père Marin Mersenne
(1588-1648)
Henry Oldenburg
(1619-1677)
http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg
http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg
Information
Hubs
6. Scientific Societies
L’Académie royale des sciences
22 December 1666
The Royal Society
28 November 1660
http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg
https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview
http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
7. Scientific Societies
L’Académie royale des sciences
22 December 1666
The Royal Society
28 November 1660
http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg
https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview
http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg
Institutionalization
of Science
8. Scientific Journals
Le journal des sçavans
5 January 1665
Philosophical Transactions
6 March 1665
http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg
http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
9. Scientific Journals
Le Journal des sçavans
5 January 1665
Philosophical Transactions
6 March 1665
http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg
http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg
Registration
Certification
Dissemination
Archiving
10. Scientific Articles: 17th to Early 19th Century
• Experiments and descriptions of the natural world
• Avoiding “fine speaking”
• Various styles of arguing
• Qualitative and personal judgements
Harmon, J.E. & Gross, A.G. (2007). The Scientific Literature. A Guided Tour. Chicago: University of Chicago Press.
11. Modern Scientific Articles
• Professionalized and highly specialized
• Increased focus on data, graphs, tables and theory
• Impersonal, technical and codified
• Style guides and gatekeeping
• Citations
• Introduction, Methods, Results and Discussion
Larivière, V., Archambault, É. & Gingras, Y. (2008). Long-term variations in the aging of scientific literature: From exponential growth to steady-state science (1900-2004). Journal of the American
Society for Information Science and Technology, 59(2), 288-296.
Sollaci, L.B. & Pereira, M.G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. Journal of the Medical Library Association, 92(3), 364-371
Sollaci&Pereira(2004,p.365)
Proportion of IMRaD adoption in medical journalsNumber of references 1900 to 2004
Larivière,Archambaul&Gingras(2008,p.293)
12. arXiv submission statistics from http://arxiv.org/stats/monthly_submissions
Larivière, V., Lozano, G.A. & Gingras, Y. (2014). Are elite journals declining? Journal of the Association for Information Science and Technology, 65(4), 649-655.
• Improved access
• Acceleration
• Collaboration
• Peer review
• Distribution of preprints
• Decreasing importance of
scientific journals
• Journal functions
• Diversification of publication
venues
• Appearance and symbolic
capital of journals unchanged
Digital Revolution
Submissions to arXiv
Share of top 1% most cited papers
Larivière,Lozano&Gingras(2014,p.652)
13. Academic Publishing Market
Larivière, V., Haustein, S. & Mongeon, P. (in press). The oligopoly of academic publishers in the digital era. PLoS ONE.
Larivière,Haustein&Mongeon(inpress)
• Aggravation of
serials crisis
• Elsevier: €3,400
subscription price,
30% increase
• Profit margins of
commercial publishers
up to 40%
• Decline of
scientific societies
as publishers
• >50% of papers owned
by five major
publishers
14. Budapest Open Access Initiative
Open Access
“immediate, free availability on the public internet, permitting
any users to read, download, copy, distribute, print, search or
link to the full text of these articles”
Budapest Open Access Initiative (2002)
Archambaultetal.(2013,p.13)
Freely available journal papers 2004 to 2011
• Gold and Green
• Libre and Gratis
• Hybrid journals
• Elsevier: $500 to 5,000
• Springer: $3,000
• Wiley: $3,000
Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Rebout, L. & Roberge, G. (2013). Proportion of Open Access Peer-Reviewed Papers at the European and World Levels 2004-2011. Report for
the European Commission. http://www.science-metrix.com/pdf/SM_EC_OA_Availability_2004-2011.pdf
15. Open Science
“opening up the research process by making all of its
outcomes, and the way in which these outcomes were
achieved, publicly available on the World Wide Web”
Kraker et al. (2011, p. 645)
Kraker, P., Leony, D., Reinhardt, W. & Beham, G. (2011). The case for an open science in technology enhanced learning. International Journal of Technology Enhanced Learning, 3(6), 643-654.
• Open Data
• Open Source
• Open Methodology
• Open Access
• Open Peer Review
16.
17. Björneborn, L. & Ingwersen, P. (2004), Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227.
Definition of
Scholarly Metrics
adaptedfrom:Björneborn&Ingwersen(2004,p.1217)
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
18. Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique.
Pritchard, P. (1927). Statistical bibliography or bibliometrics? Journal of Documentation, 25, 348-349..
Definition of
Scholarly Metrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
“La «Bibliometrie» sera
la partie définie de la
Bibliologie qui s'occupe
de la mesure ou quantité
appliquée aux livres.”
“the application of
mathematics and
statistical methods to
books and other media
of communication”
Pritchard (1969, p. 348)
Otlet (1934, p. 14)
19. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press.
Rousseau, R. & Ye, F. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 10–12. doi:10.1007/s11434-013-5939-3
informetrics
scientometrics
cybermetrics
webometrics altmetrics
“study and use of
scholarly impact
measures based on
activity in online tools
and environments”
“a good idea but a
bad name”
Rousseau & Ye (2013, p. 2)
Priem (2014, p. 266)
bibliometrics
Definition of
Scholarly Metrics
20. Definition of
Scholarly Metrics
informetrics
scientometrics
bibliometrics
cybermetrics
webometrics altmetrics
scholarly metrics
Scholarly metrics are
metrics based on acts
and events (e.g.,
viewing, reading, saving,
diffusing, mentioning,
citing, reusing, modifying)
related to scholarly
documents (e.g.,
papers, blog posts,
datasets, code, notes)
or scholarly agents
(e.g., researchers,
universities, journals).
21. Gross, P.L.K. & Gross, E.M. (1927). College libraries and chemical education. Science, 66(1713), 385-389..
Citation analysis for objective collection management
Bibliometrics for
Library Mangement
Gross&Gross,1927,p.386
Journals cited in the Journal of the American Chemical Society 1926
22. Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108-111.
Bibliometrics for
Information Retrieval
Retrieval tool to handle information overload
• Institute for Scientific Information
• Science Citation Index
• Source Author Index
• Citation Index
“It would not be excessive to demand that the
thorough scholar check all papers that have
cited or criticized such papers, if they could be
located quickly. The citation index makes this
check practicable.”
Garfield, 1955, p. 108
23. Derek J. de Solla Price
Science since Babylon (1961)
Little Science – Big Science (1963)
Bibliometrics for
Sociology of Science
Price, D. J. d. S. (1961). Science Since Babylon. New Haven / London: Yale University Press,
Price, D. J. d. S. (1963). Little Science, Big Science. New York: Columbia University Press.
24. Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623.
Robert K. Merton
• Social norms of science
• Communalism
• Universalism
• Disinterestedness
• Organized skepticism
• Matthew effect
Bibliometrics for
Sociology of Science
“symbolically, [the reference] registers in the
enduring archives the intellectual property of
the acknowledged source by providing a
pellet of peer recognition of the knowledge
claim” Merton (1988, p. 621)
25. • Performance measurement and policy instrument
• Commercialization
Bibliometrics for
Research Evaluation
“When used properly, bibliometric indicators can provide a
‘monitoring device’ for university research-management and
science policy. They enable research policy-makers to ask
relevant questions of researchers on their scientific
performance, in order to find explanations of the bibliometric
results in terms of factors relevant to policy.”
Moed et al. (1985, p. 131)
Moed, H.F., Burger, W.J.M., Frankfort, J.G, van Raan, A.F.J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131-149.
26. • Oversimplification
publications = productivity
citations = impact
• Uninformed use and misuse
Impact Factor
h-index
• Adverse effects
“salami” publishing
honorary authorship
self-citations
citation cartels
Bibliometrics for
Research Evaluation
Hvistendahl, M. (2013). China’s publication bazaar. Science, 342(6162), 1035-1039.
van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for ‘citation stacking’, Nature, 500(7464), 510-511.
27. Criticism against current form of research evaluation
Ø Alternative forms of research ouput
Ø Alternative use and visibility of publications
Altmetrics
“We rely on filters to make sense of the scholarly literature, but the
narrow, traditional filters are being swamped. However, the growth of
new, online scholarly tools allows us to make new filters; these
altmetrics reflect the broad, rapid impact of scholarship in this
burgeoning ecosystem.”
…
Priem et al. (2010)
Information overload
…
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-metrics: a manifesto. October. Retrieved from http://altmetrics.org/manifesto/
Piwowar, H. A. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.
28. Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215.
(for Mendeley reader counts only: 2010-2012 PubMed/WoS papers)
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
(for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
Altmetrics
What do we know?
%
29. Altmetrics
What do we know?
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215.
(for Mendeley reader counts only: 2010-2012 PubMed/WoS papers)
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
(for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
%
30. Altmetrics
What do we know?
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215.
(for Mendeley reader counts only: 2010-2012 PubMed/WoS papers)
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
(for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)
%
31. Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
Altmetrics
What do we know?
Biomedical & Health Sciences
Social Sciences & Humanities
Life & Earth Sciences
Natural Sciences & Engineering
Mathematics & Computer Science
Twittercoverage
32. Documenttype
Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.
Altmetrics
What do we know?
Biomedical & Health Sciences
Social Sciences & Humanities
Life & Earth Sciences
Natural Sciences & Engineering
Mathematics & Computer Science
Twittercoverage
39. Meaning of metrics
Opportunities
• Heterogenous users
• Diverse motivations
Challenges
• Understanding underlying processes
• Determining the meaning of metrics
42. Meaning of metrics
Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (in press). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology.
Bertin,Atanassova,Gingras&Larivière(inpress,p.13)
Distribution of references along the IMRaD structure
Citing in a
journal article
43. Meaning of Metrics
Haustein, S., Bowman, T.D. & Costas, R. (in press). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In Sugimoto , C.R. (Ed.). Theories of
Informetrics: A Festschrift in Honor of Blaise Cronin.
Acts leading to (online) events used for metrics
Haustein,Bowman&Costas(inpress,p.13)
RESEARCH
OBJECT
44.
45. Opportunities
• Detailed life cycle of scientific output
• Fine-grained indicators and adequate benchmarks
Challenges
• Versions of research output
• Publication dates
Time Analysis
46. Time Analysis
Journal article
• Submitted manuscript
• Revised manuscript
• Accepted manuscript
• Version of Record
• Online publication
• Journal issue
• Online date
• Issue month
Ø Adjusting indicators
3 March 2014
15 July 2014
21 January 2015
February 2015
48. Time Analysis
Haustein, S., Bowman, T.D. & Costas, R. (in press). When is an article actually published? An analysis of online availability, publication, and indexation dates. Proceedings of the 15th International
Conference on Scientometrics and Informetrics, Istanbul, Turkey.
Tweets before publication?
49.
50. Communities of Attention
Opportunity
• Differentiating between types of use
• Measuring social impact
Challenges
• Determining engagement
• Identifying users
51. Level of engagement on Twitter
Communities of Attention
!Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter.
Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456
52. Communities of Attention
Automated Twitter bots
Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter.
Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456
tweeting preprints
from arXiv subject
area
tweeting topic-
relevant papers
selection of
interesting papers
53. Communities of Attention
exposure
engagement
median dissimilarity with paper title
mediannumberoffollowers
influencers /
brokers
orators /
discussing
disseminators /
mumblers
broadcasters
tweet text differs
from paper title
tweet text is identical
to paper title
few followers many followers
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
54. Communities of Attention
708 of 125,083 users (0.6%) tweeting
WoS papers published in 2012 (>100)
Node size represents number of papers
Network of users tweeting the same papers
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
55. Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
56. Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
57. Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
58. Communities of Attention
Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015,
Toronto, Canada.
61. Stefanie Haustein
Thank you for your attention!
Questions?
stefanie.haustein@umontreal.ca
@stefhaustein
crc.ebsi.umontreal.ca/sloan
Thank you for your attention!