Presentation at APE2014

1. Tweets and Mendeley readers
Two different types of article level metrics
Stefanie Haustein
stefanie.haustein@umontreal.ca
@stefhaustein

2. Overview
• Altmetrics
• increasing use
• meaning?
• Aim of the studies
• Data sets and methods
• Results
• documents
• correlations
• disciplines
• Conclusions & outlook

3. Altmetrics: increasing use
• social media activity around scholarly articles growing by
5% to 10% per month (Adie & Roe, 2013)
• Mendeley and Twitter largest altmetrics sources
• Mendeley
• 521 million bookmarks
• 2.7 million users
• 32% increase of users from 09/2012 to 09/2013
• Twitter
• 500 million tweets per day
• 230 million active users
• 39% increase of users from 09/2012 to 09/2013
Adie, E. & Roe, W. (2013). Altmetric: Enriching Scholarly Content with Article-level Discussion and Metrics. Learned Publishing, 26(1), 11-17.
Mendeley statistics based on monthly user counts from 10/2010 to 01/2014 on the Mendeley website accessed through the Internet Archive
Twitter statistics: https://business.twitter.com/whos-twitter and http://www.sec.gov/Archives/edgar/data/1418091/000119312513400028/d564001ds1a.htm

4. Altmetrics: meaning?
• ultimate goals
• similar to but more timely than citations
Ø predicting scientific impact
• different, broader impact than captured by citations
Ø measuring societal impact
• impact of various outputs
Ø “value all research products”
Piwowar (2013)
Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159.

5. Altmetrics: meaning?
• Altmetrics are “representing very different things”
(Lin & Fenner, 2013)
• unclear what exactly they measure:
• scientific impact
• social impact
• “buzz”
Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards
Quarterly, 25(2), 20-26.

6. Altmetrics: meaning?
ad-hoc
classifications
need to be
supported
by research

7. Altmetrics: meaning?
scientist on
Twitter tweeting
scientific paper
in non-scholarly
manner:
• scientific impact?
• social impact?
• buzz?

8. Altmetrics: meaning?

9. Aim of the studies
• providing empirical evidence of Mendeley reader counts
and tweets of scholarly documents for a large data set
• generate knowledge about factors influencing popularity of
scholarly documents on Mendeley and Twitter
• analyzing the following research questions:
•
•
•
•
What is the relationship between social-media and citation counts?
How do social-media metrics differ?
Which papers are highly tweeted or highly bookmarked?
How do these aspects differ across scientific disciplines?
Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (2014).
Tweeting Biomedicine: An Analysis of Tweets and Citations in the Biomedical Literature.
Journal of the Association for Information Sciences and Technology.
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs.
Mendeley readers: How do these two social media metrics differ? IT-Information Technology.

10. Aim of the studies
• large-scale analysis of tweets and Mendeley readers of
biomedical papers
• Twitter and Mendeley coverage
• Twitter and Mendeley user rates
• correlation with citations
• discovering differences between:
• documents
• disciplines & specialties
Ø providing an empirical framework to compare coverage,
correlations and user rates

11. Data sets & methods
• 1.4 million PubMed papers covered by WoS
• publication years: 2010-2012
• document types: articles & reviews
• matching of WoS and PubMed
• tweet counts collected by Altmetric.com
• collection based on PMID, DOI, URL
• matching WoS via PMID
• Mendeley readership data collected via API
• matching title and author names
• journal-based matching of NSF classification

12. Data sets & methods: framework

13. Data sets & methods: age biases
Current biases influencing correlation coefficients
Ø compare documents of similar age
Ø normalize for age differences

14. Results: documents
• Twitter coverage is quite low but increasing
• correlation between tweets and citations is very low
Publication
year
Twitter
coverage
Papers
(T≥1)
Spearman's ρ
Mean
Median
Maximum
T2010
C2010
2.4%
13,763
.104**
2.1
18.3
1
7
237
3,922
T2011
C2011
10.9%
63,801
.183**
2.8
5.7
1
2
963
2,300
T2012
C2012
20.4%
57,365
.110**
2.3
1.3
1
0
477
234
9.4%
134,929
.114**
2.5
5.1
1
1
963
3,922
T2010-2012
C2010-2012

15. Results: documents
Top 10 tweeted documents:
catastrophe & topical / web & social media / curious story
scientific discovery / health implication / scholarly community
Article
Journal
C
T
Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients
is associated with exposure to low-dose irradiation
PNAS
9
963
Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the
Fukushima nuclear accident
PNAS
30
639
Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at
Our Fingertips
Science
11
558
Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator
Journal of Physical
Chemistry A
--
549
Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes
Cutis
--
477
Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life
expectancy: a prospective cohort study
Lancet
51
419
Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations
Journal of Sexual
Medicine
--
392
Newman & Feldman (2011). Copyright and Open Access at the Bedside
New England
Journal of Medicine
3
332
Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells
Science
5
323
Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA
receptor expression in a mouse via the vagus nerve
PNAS
31
297

16. Results: correlations
PubMed papers covered by Web of Science (PY=2011)
Spearman correlations between citations (C), Mendeley readers (R) and tweets (T) for all papers published in
2011 (A, n=586,600), for papers with respectively at least one citation (B, n=410,722), one Mendeley reader (C,
n=390,190) or one tweet (D, n=63,800), one Mendeley reader and one tweet (E, n=45,229) and one citation, one
Mendeley reader and one tweet (F, n=36,068). All results are significant at the 0.01 level (two-tailed).

17. Results: disciplines
PubMed papers covered by Web of Science 2010-2012

18. Altmetrics: disciplinary biases
x-axis:
coverage of
specialty on
platform
y-axis:
correlation
between social
media counts
and citations
bubble size:
intensity of use
based on mean
social media
count rate

19. Results: disciplines
General Biomedical Research papers 2011
Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**),
bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A)
and read (B) papers are labeled showing the first author.

20. Results: disciplines
Public Health papers 2011
Scatterplot of number of citations and number of tweets (A, ρ=0.074**) and Mendeley readers (B, ρ=0.351**)
for papers published in Public Health in 2011. The respective three most tweeted (A) and read (B) papers are
labeled showing the first author.

21. Conclusions & outlook
• uptake, usage intensity and correlations differ between
disciplines and research fields
Ø social media counts of papers from different fields are not
directly comparable
• citations, Mendeley readers and tweets reflect different
kind of impact on different social groups
• Mendeley seems to mirror use of a broader but still academic
audience, largely students and postdocs
• Twitter seems to reflect the popularity among a general public
and represents a mix of societal impact, scientific discussion
and buzz
Ø the number of Mendeley readers and tweets are two
distinct social media metrics

22. Conclusions & outlook
• before applying social media counts in information
retrieval and research evaluation further research is
needed:
Ø identifying different factors influencing popularity of
scholarly documents on social media
Ø analyzing uptake and usage intensity in various disciplines
Ø differentiating between audiences and engagements

23. Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting
biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the
Association for Information Sciences and Technology.
Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (submitted). Tweets vs. Mendeley
readers: How do these two social media metrics differ? IT-Information Technology.
Thank you for your attention!
Questions?
Stefanie Haustein
stefanie.haustein@umontreal.ca
@stefhaustein