This document summarizes research measuring academic influence by analyzing citations. It discusses:
1) Background on citation analysis and its limitations in counting all citations equally;
2) An experiment collecting "influential" citations from papers to build a dataset;
3) Analyzing paper-reference pairs using machine learning classifiers trained on features like citation counts, context similarity, and position;
4) Proposing "influence-primed measures" that weight citations based on frequency to better measure influence, like an influence-primed h-index and impact factor. The researchers conclude influence can be measured by counting more relevant citations.
Measuring Academic Influence Not All Citations Equal
1. Xiaodan Zhu and PeterTurney
National Research Council Canada
Daniel Lemire
TELUQ, Université du Québec Montréal
AndreVellino
School of Information Studies,
University of Ottawa, Ottawa
Measuring Academic Influence:
Not All Citations Are Equal
2. Overview
— Some background in CitationAnalysis
— What we tried to do and why
— How we did it
— What the results were
— What the implications are
3. What is Citation Analysis
Citation analysis refers to the collection of methods for measuring
the importance of scholars, journals and institutions by counting
citations in a graph of references in the published literature.
…
…
… …
…
…
4. Why Do Citation Analysis?
— Reason # 1: Because it generates measurable quantities!
“Since we can’t really measure what
interests us, we begin to be interested
in what we can measure”
JoelWestheimer
Professor of Education
University of Ottawa
5. Uses for Citation Measures
— For Readers
— To evaluate the quality of articles / journals
— For Universities
— To evaluate the productivity of academics
— To help in tenure and promotion decisions
— For Journals
— To attract authors to publish
— For Libraries
— To make collections / acquisition decisions
— To make automated recommendations to users
6. How Are Citations Counted?
— Add 1 for every new occurrence of a cited article
— Sum the results
— Average per article & / or CountTotal # of citations
Problems
— Self citations!
— No measure of quality of citing source
— May be skewed by a small number of highly cited items
— Easy to “game” by tricking Google Scholar
— viz. Ike Inktare h-index = 94 – Einstein h-index = 84
7. h-index
— Jorge Hirsch (PNAS, 2005) defined the h-index:
— Attempts to measure both the productivity and impact of the
author’s published work
— An author has index h if h of their N papers have at least h citations
each, and the other (N − h) papers have at most h citations each.
8. Some Criticisms of the h-index
— The h-index does not account for the number of authors or the order of
the authors of a paper.
— Cannot use the h-index to compare authors in different fields
— Young researchers with as yet short careers are at a built-in disadvantage
over older researchers
— Constrained by the total number of publications
— 10 papers each w/ 100 citations each = 10 papers w/ 10 citation each
“[h-index] captures a small amount of information
about the distribution of a scientist's citations [and] loses crucial
information that is essential for the assessment of research.”
Adler, R., Ewing, J.Taylor, P. Citation statistics.
A report from the International Mathematical Union.
http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
9. Journal Impact factor (IF)
— Invented by Eugene Garfield in 1955 to identify journals for
Science Citation Index
— Definition:
Total Citations (2 preceding years )
Total Articles (2 preceding years )
=JIF
i.e. the impact factor of a journal is the average number
of citations to those papers that were published during
the two preceding years
¨ e.g. the number of times articles published in 2001 and 2002
were cited by indexed journals during 2003 / the total number
of items published in 2001 and 2002
10. Some Criticisms of Impact Factor
— Letters or editorials in some journals (e.g. Nature) are often cited
(and counted) in “Total Citations” (numerator) but not in “Total
Articles”
— 2-year window not applicable in many fields (e.g. in Math 90% of
citations fall outside the 2-year window)
— IF varies considerably across disciplines (Math has an average of
0.9 citation per article, Life Sciences have an average of 6.2)
“Using the impact factor alone to judge a journal is
like using weight alone to judge a person's health.”
Adler, R., Ewing, J.Taylor, P. Citation statistics.
A report from the International Mathematical Union.
http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf
12. — As early as 1965 Garfield identified 15 different reasons for
citing
— giving credit for related work
— correcting a work
— criticizing previous work
— Many attempts since to categorize citations
One Big Assumption
All citations should count equally!
13. Citation Typing Ontology (CiTO)
Here are first 21 of the 91 citation types in CiTO
http://imageweb.zoo.ox.ac.uk/pub/2008/plospaper/latest/#refs
Example of semantically annotated article using CiTO:
14. Our Objective
— Solve a binary classification problem:
Given a Paper-Reference (P-R) pair, does
P-R belong to the class “R is highly
influential for P” or not.
Our Method
— Apply Machine Learning methods to train a computer to
recognize “Highly Influential Reference” from examples
15. Step 1 – Data Collection
We believe that most papers are based on 1, 2, 3 or 4
essential references. By an essential reference, we mean a
reference that was highly influential or inspirational for the
core ideas in your paper; that is, a reference that inspired or
strongly influenced your new algorithm, your experimental
design, or your choice of a research problem. Other
references merely support the work.
16. We asked for
— Title of your paper (research papers only; no surveys)
— The essential references does your paper build?
We got
— 100 papers
— 322 “influential” references
— i.e. 3.2 “influential references” per article
— Each paper
— Contained ~ 31 references in the References section
— Cited ~ 54 references in the body of the paper
— i.e. each reverence was cited an average of 1.7 times per paper
17. The Problem
— The 100 papers yield 3143 paper-reference pairs
— The authors have selected ~320 paper-reference pairs
— Algorithmically: to accurately select those 320 from the 3142
18. Paper – Reference Analysis
— OpenNLP used to detect sentence boundaries and tokenize.
— ParsCit to parse the papers.
— ParsCit is an open-source package for parsing references and
document structure in scientific papers.
— Regular expressions to capture citation occurrences in paper
bodies that were not detected by ParsCit.
20. We Looked at 5 Classes of Features
1. Count-based features
2. Similarity-based features
3. Context-based features
4. Position-based features
5. Miscellaneous features
21. Count Based Features
— Total number of times a paper is referenced in the citing paper
— The number of different sections in which a given reference appears
— Number of times a paper is referenced in the
— “Related” section
— “Introduction” section
— “Core” sections (all sections excluding “Related”,“Introduction”,
“Acknowledgements”,“Conclusion” and “FutureWork”
— The number of different sections in which a reference appears
23. Citing Context
— When an article is cited, the linguistic context in which the
article is cited is considered as saying something about the
cited article.
e.g.
“Like Moravcsik and Murugesan (1975),we are concerned
about the side effects of counting insignificant references”
25. Other Context Based Features
— Authors explicitly mentioned in citation context?
— Citation alone [4] or with others [3,4,5]
— If “with others” is it first? (e.g.“[3]” is first in “[3,4,5]”)
Using pre-defined word-lists, is the lexical content of a citation
— “relevant” [likewise,influential,inspiring useful….]
— “new” [recently,latest,current,improved…]
— “extreme” [greatly,intensely,acutely,almighty,awfully]
— “comparative” [easy,easier,easiest,strong,stronger…]
26. Lexical Context Features
Using a lexicon of 114,271 words obtained from the General
Inquirer Lexicon (11,788 words) extended w/Wordnet +
Turney and LittmanAlgorithm,
— Count the number of words labeled
— “Strong”
— “Positive”
— “Evaluative”
Also, sentiment analysis with a different lexicon gave us
— Presence / absence of “Emotion” (Joy, Sadness,Anger, Fear, etc.)
— “Positive” / “Negative”
27. Position Based Features
Where does the citation occur?
— Citation appears at the beginning of a sentence? (Y/ N)
— Citation appears at the end of a sentence? (Y/N)
— Where are the sentence(s) in which the citation(s) occur(s)
e.g.
— 0 (First sentence) to 1 (Last sentence)
— distance from the mean of occurrences of all citations
29. Top 7 Features: 4 “counts”, 3 “similarity”
Counts in Paper
Counts in Sections
Counts in Core Section
Title-Abstract Similarity
Counts in Intro Section
Title-Core Similarity
Title-Intro Similarity
32. hip-index
— Each occurrence of a citation of paper R by paper C = 1
— hip-index (h-influence-primed) index for an author is the
largest number h such that at least h of the author's papers
have an influence-primed citation count of at least h.
33. Examples
hip-index = 5
h-index = 2
cited 3 times by C1 = 9
cited 2 times by C2 = 4
cited 2 times by C3 = 4
cited 2 times by C4 = 4
R3 – cited 3 times by C5 = 9
R4 – cited 3 times by C6 = 9
R5 – cited 3 times by C7 = 9
R6 – cited 2 times by C8 = 4
R7 – cited 1 times by C9 = 1
13
8
9
9
9
4
1
hip-index = 3
h-index = 2
cited 2 times by C1 = 4
cited 1 times by C2 = 1
cited 2 times by C3 = 4
cited 1 times by C4 = 1
R3 – cited 2 times by C5 = 4
R4 – cited 1 times by C6 = 1
R5 – cited 1 times by C7 = 1
R6 – cited 1 times by C8 = 1
R7 – cited 1 times by C9 = 1
5
5
4
1
1
1
1
R1
R2
R1
R2
34. Using hip-index to Predict ACM Fellows
— Used the citation network constructed from
— ~ 20,000 papers in theAssociation for Computational Linguistics
Anthology
— Calculated the h-index ofACL Fellows
— Calculated the hip-index ofACL Fellows
— Compared the precision of h-index and hip-index
— the number ofACL Fellows in the top N divided by N
36. Conclusions
— We can throw away h-index and Impact Factor etc. completely
OR we can try to improve them by counting citations more
relevantly
— A measure of academic influence for a citation is possible and
— It is easy to compute to a first approximation – merely count
their frequency
— Apply the influence-primed weights on citation graphs to
compute
— Influence-primed Impact Factor, g-index etc.