Continued citation of bad science and what we can do about it--2021-04-20

Continued Citation of Bad Science and
What We Can Do About It
Jodi Schneider
School of Information Sciences
University of Illinois at Urbana-Champaign
jodi@illinois.edu
Twitter:@jschneider
infoqualitylab.org
GW - CNHS Informatics Seminar
2021-04-20

What is retraction?
“Retraction is a mechanism for correcting the literature and
alerting readers to articles that contain such seriously flawed or
erroneous content or data that their findings and conclusions
cannot be relied upon.”
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019
https://doi.org/10.24318/cope.2019.1.4

What is retraction?
“Retraction is a mechanism for correcting the literature and
alerting readers to articles that contain such seriously flawed or
erroneous content or data that their findings and conclusions
cannot be relied upon.”
“Prompt retraction should minimise the number of researchers
who cite the erroneous work, act on its findings, or draw
incorrect conclusions such as from ‘double counting’ redundant
publications in meta-analyses or similar instances."

Retracted papers still get cited & used

Retracted papers still get cited & used
doi:10.1126/science.abg5806

Plan for today
• Case study of a retracted paper, analyzing text & network data
• Large-scale study of almost 50,000 citation contexts
• Finding knowledge dependencies using argumentation theory
& citation context analysis
• Stakeholder engagement to address the problem

Case study of a retracted paper

Retracted October 2008 because an author falsified data

This RCT paper is retracted…. But how would you know?

Retraction doesn’t stop citation.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years
after it was retracted for falsifying data.” In Scientometrics. http://doi.org/10.1007/s11192-020-03631-1

Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years
after it was retracted for falsifying data.” In Scientometrics. http://doi.org/10.1007/s11192-020-03631-1
5 out of 112 post-retraction citations
mention the retraction.
The remaining 107 use the paper –
and its faked data – as normal
science.

Since its 2008 retraction, this RCT paper has been cited over 100 times.
Only 4% (5/112) mention the retraction.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years after it was retracted for
falsifying data.” In Scientometrics. http://doi.org/10.1007/s11192-020-03631-1

Only 5 mention the retraction
2008
32 direct citations
63 indirect published 2008
2014 2019
114 direct citations
148 direct citations
Only 2 mention the retraction
retraction
RCT Retracted:
Author falsified data
07
Paper (black)
n articles (blue): 22
on articles (red): +25
2008
Matsuyama Paper (black)
# of first−generation articles (blue): 32
# of second−generation articles (red): +63
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial
report, eleven years after it was retracted for falsifying data.” In Scientometrics. http://doi.org/10.1007/s11192-020-03631-1

Described the Matsuyama paper’s methods and/or results (but not the retraction):
35 direct citations 2010-2019 (blue squares)
Cited by 161 articles (red squares)
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics.
http://doi.org/10.1007/s11192-020-03631-1
Specific citation 2010−2019

“Health benefits of flaxseed” (Fitzpatrick, 2011)
Government research bulletin from Nepal (Jha, 2016)
Spread of misinformation to a second generation
which cites the retracted paper as “evidence”
of the anti-inflammatory impact of flax.
“Whilst it is true that very little
ALA converts to the long chain
polyunsaturated omega-3 found in
marine oils, it does have beneficial
effects itself (Fitzpatrick, 2011).”
cites a book chapter
http://doi.org/10.1007/s11192-020-03631-1

An Irish nutritional support shop
recommends n-3 fats to athletes
(Healthy fats, fish oils & omega-
3 supplementation, 2017)
which cites the retracted paper as having demonstrated “Improved 6 min walk test,
decreased leukotriene B4 level, TNF-alpha, IL-8 [91]” (i.e., the faked data for which the
Matsuyama paper was retracted)
“During periods of illness, this may help promote
recovery and faster return to training. Interestingly, n-3
fats are sometimes provided to COPD patients (severe
airway damage and breathing difficulties) and prior to
surgery in order to support the immune system and
speed recovery by helping to control inflammation and
infection, and repair damaged cells17.”
“Immunologic impact of nutrient depletion in chronic obstructive pulmonary disease”,
(Herzog & Cunningham-Rundles, 2011)
cites an article
http://doi.org/10.1007/s11192-020-03631-1

A pre-clinical study on lung
repair following dust exposure
(Nordgren et al., 2018)
where the only cited “evidence” for the effect of n-3 PUFAs on inflammation in COPD
came from the retracted Matsuyama paper.
“Furthermore, studies reveal diets high in omega-3
polyunsaturated fatty acids (n-3 PUFA) may be
beneficial in inflammatory lung conditions, including
asthma and COPD (17).”
“Beneficial effects of n-3 PUFA on chronic airway inflammatory diseases” (Giudetti
and Cagnazzo, 2012)
cites this article to motivate its work:
http://doi.org/10.1007/s11192-020-03631-1

Difficult to get to the retraction notice via database
search
• We tested 8 databases: Only 1 (EMBASE) had a working link to the retraction notice!
• Linking errors give the impression that the retraction notice doesn’t exist!
http://doi.org/10.1007/s11192-020-03631-1

Large-scale study of citations to retracted papers

Large-scale study of citations in biomedical papers
Started with:
• Retracted papers: All 7813 retracted papers in PubMed as of August
2020
• Citation contexts: All papers in PubMed Central open access XML subset
as of May 2019 (Hsiao, T.-K., & Torvik, V. I., in preparation).
Our data set:
49,236 citation contexts citing 4611 retracted papers
Included 13,364 post-retraction citation contexts citing 2765
retracted papers
Tzu-Kun Hsiao & Jodi Schneider. Continued Use of Retracted Papers- Temporal Trends in Citations and (Lack of) Awareness of
Retractions Shown in Citation Contexts in Biomedicine. Working paper under review
https://osf.io/4jexb/?view_only=ab8e49facd5d4372b52d1419af8b9651

Large-scale study of citations in biomedical papers
Examples from the 13,364 post-retraction citations in our data set:
• A clinico-histopathologic study in rabbits confirmed that PRP treatment
can achieve a faster wound healing rate |B8|.
• However, to date, only one human study has demonstrated an induction
of SIRT1 mRNA level in PBMCs |sbref41|.

Retracted papers are not cited differently from
non-retracted papers

Few authors show awareness of retraction
• 613 citation contexts out of 13,000+ (4.6%) post-retraction mentioned
the retraction
• These typically address previous accomplishments or historical events.
• May provide underlying data for the study (e.g. study of ethics, retraction, etc.)

Few authors show awareness of retraction
• 95% of post-retraction citations do not show awareness of the
retraction

Intentional citation of retracted papers
Previous accomplishment:
Initial reports indicated that CAF-1 binds H3_H4 in a
dimeric conformation in vivo (27,84,85) and that CAF-
1 could form homodimers (86). Later reports have
suggested that CAF-1 is monomeric (55,57,58),
and also that a tetrameric histone conformation
might exist in the complex ([22941638,
Rnotice:28934509]). However, the most recent
reports have proposed an updated and unified model
for how CAF-1 interacts with histones.

Background event:
In 1998, a Lancet article [9500320] was widely
interpreted as suggesting that MMR was linked with
autism and bowel problems. Although the paper was
eventually discredited and retracted by the Lancet
in 2010 [Rnotice:20137807], widespread adverse
media coverage in the intervening years led to a
dramatic fall in MMR uptake, to a low of 80% in
England (of the first dose by age 2) in 2003.

Exclusion rationale:
Moreover, we excluded 2
retracted articles
[20412149, 19352169].

Notify retraction included:
The findings of one of our sub-
analyses, that ACE inhibitors did
not influence leg outcomes is in
apparent contradiction to a
previous meta-analysis
which reported that ACE
inhibitors improved
walking ability in patients
with IC[44]. It should be
noted that the prior meta-
analysis included two trials
which have been
subsequently
retracted[23385271,
16670135] and positive data
from these studies may have
contributed to this difference in
findings.

(613 citation contexts)

InterAnnotator Agreement:
Would a second annotator choose the same categories for a
random sample of 100 of the 613 citation contexts?
• The Kappa coefficient of the inter-coder agreement was .49 – moderate
agreement.
• Complexity of the task?
• Difficulty in applying the decision map? (e.g., frequently need to look at the
whole paper)
• Experience of the annotator? Scientometrics grad student, non-native speaker.
• Most common problems:
• Notify retraction included all annotated as Previous accomplishment
• Previous accomplishment vs. Background event
• Example of problematic science

Finding knowledge dependencies by using
argumentation theory and citation context analysis

Fu, Yuanxi, Jodi Schneider. 2020. “Towards knowledge maintenance in scholarly digital libraries with
keystone citations.” In JCDL 2020, 217–226. doi:10.1145/3383583.3398514
When a paper gets retracted, does that impact
the validity of a paper that cites it?

It depends on the argument structure!
Retraction Watch: https://retractionwatch.com/2020/12/09/authors-of-
meta-analysis-on-heart-disease-retract-it-when-they-realize-a-nejm-
reference-had-been-retracted/

Conclusions of a systematic reviews and meta-
analyses depend on EACH article they synthesize!
Retraction Watch: https://retractionwatch.com/2020/12/09/authors-of-
meta-analysis-on-heart-disease-retract-it-when-they-realize-a-nejm-
reference-had-been-retracted/
Fu, Yuanxi, Jodi Schneider. 2020. “Towards knowledge maintenance in
scholarly digital libraries with keystone citations.” In JCDL 2020,
217–226. doi:10.1145/3383583.3398514

Citing Article Cited Article
Cites
Support arguments
Under my keystone citation
framework:
1) A scientific research paper puts forward
at least one main finding, along with a
logical argument, giving reasons and
evidence to support the main finding.
2) The main finding is accepted (or not) on
the basis of the logical argument.
3) Evidence from earlier literature may be
incorporated into the argument by citing a
paper and presenting it as support, using
a citation context.
Citation Contexts
Arguments
modeled

Step 4
Flag those articles
that are potentially
impacted
Workflow for Flagging Problematic Citations
Step 2
The domain expert
develops a list of screening
questions
Step 3
Experts/non-experts/text
mining tools screen target
articles using the checklist
Step 1
The domain expert develops
a generalized argument
model

https://infoqualitylab.org/projects/risrs2020/

Reducing the Inadvertent Spread of Retracted Science
2020-
2021
Jodi Schneider jodi@illinois.edu
Information Quality Lab https://infoqualitylab.org/
Collaboration across diverse stakeholders: funders, editors, peer reviewers, authors,
publishers, database providers, research integrity officers, science journalists
Advisory Board
National Center
for Professional
and Research Ethics
Retraction Watch
JAMA
Annettte Flanagin
RN, MA, FAAN
Ivan Oransky, MD
C.K. Gunsalus, JD
London School of
Economics and Political
Science
Daniele Fanelli, PhD

Jodi Schneider MT Campbell Nathan Woods Yuanxi Fu Tzu-Kun (Esther) Hsaio
Randi Proescholdt Vivien Yip Yoss Arianlou Halle Burns
RISRS 2020 Team

The project overall is trying to address these 4 research questions:
1.What is the actual harm associated with retracted research?
2.What are the intervention points for stopping the spread of retraction? Which
gatekeepers can intervene and/or disseminate retraction status?
3.What are the classes of retracted papers? (What classes of retracted papers
can be considered citable, and in what context?)
4.What are the impediments to open access dissemination of retraction statuses
and retraction notices?
Project Motivation

Project Timeline

● Involved about 70 people from across scholarly communication
● Interviewed 50 people from roles in research integrity, technology,
academia, libraries, publishing, journalism
● Held 3 4-hour workshops each engaging 30+ people
○ Day 1: Listening and Learning
○ Day 2: Collaborative Agenda Setting
○ Day 3: Commitment to Action & The Way Forward
Stakeholder Engagement To Date

● Initial recommendations developed through discussion & survey
● Project website with web bibliography
● Research outcomes
○ Bibliometrics & Research Assessment Symposium 2020 talk
○ SIGMET 2020 workshop poster on data quality
○ Scientometrics paper on case study, "Continued post-retraction citation of a
fraudulent clinical trial report, 11 years after it was retracted for falsifying data”
● Two industry conference presentations scheduled
○ NISO Plus 2021: Thursday, February 25, 11:30 to 12:45 PM Eastern
○ Society for Scholarly Publishing 2021, May 26
Dissemination To Date

Misinformation and truth: from fake news to
retractions to preprints
NISO Plus 2021
Thursday, February 25, 11:30 to
12:45 PM Eastern
https://sched.co/fMnP

● A coherent, actionable set of recommendations for next steps
● A white paper with the recommendations & context (currently a preprint)
● Backgrounder to White House Office of Science & Technology Policy
● Dissemination of recommendations to professional organizations
● More research publications (citation analysis under review by QSS,
lit review to be submitted to ASIST)
● Proposals for future research
● Collaborations to start achieving the recommended outcomes
Outputs To Come - by July 2021

1. Develop a systematic cross-industry approach to ensure the public
and timely availability of high-quality, consistent information about
retractions.
2. Recommend a taxonomy of retraction categories/classifications
and corresponding retraction metadata that can be adopted by all
stakeholders.
3. Develop best practices for coordinating the retraction process.
4. Educate stakeholders about retraction and pre- and post-
publication stewardship of the scholarly record.
Draft Recommendations v2

Bibliography
Cheng, Yi-Yun, Nikolaus Parulian, Tzu-Kun Hsiao, Ly Dinh, Janina Sarol, Jodi Schneider. 2019. “ReTracker: actively and automatically matching retraction metadata in Zotero.”
In ASIS&T Annual Meeting, 56(1): 372–376. doi:10.1002/pra2.32
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019 doi:10.24318/cope.2019.1.4
Dinh, Ly, M. Janina Sarol, Yi-Yun Cheng, Tzu-Kun Hsiao, Nikolaus Parulian, Jodi Schneider. “Systematic examination of Pre- and Post-Retraction Citations.” In ASIS&T Annual
Meeting, 56(1): 390–394. doi:10.1002/pra2.35
Fu, Yuanxi, Jodi Schneider. 2020. “Towards knowledge maintenance in scholarly digital libraries with keystone citations.” In JCDL 2020, 217–226.
doi:10.1145/3383583.3398514
Harris, Richard. 2020 March 26. “In Defense Of Coronavirus Testing Strategy, Administration Cited Retracted Study.” NPR. https://www.npr.org/sections/health-
shots/2020/03/26/822084429/in-defense-of-coronavirus-testing-strategy-administration-cited-retracted-study
Piller, C. (2021). Many scientists citing two scandalous COVID-19 papers ignore their retractions. Science.
Price, Gary. 2019 June 12. InfoDocket. Zotero and Retraction Watch Collaborate on New Service (Beta) That Notifies Users of Article Retractions in Their Personal Zotero
Libraries. https://www.infodocket.com/2019/06/12/zotero-and-retraction-watch-collaborate-on-new-service-beta-that-notifies-users-of-article-retractions-in-their-
personal-zotero-libraries/
Proescholdt, Randi & Jodi Schneider. 2020. Retracted Papers with Inconsistent Document Type Labeling in PubMed, Scopus, and Web of Science METRICS 2020 at ASIS&T. Poster presented
October 22, 2020 at the SIGMET METRICS 2020 workshop at ASIS&T 2020. http://jodischneider.com/pubs/sigmet2020.pdf
Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda workshop: https://infoqualitylab.org/projects/risrs2020/
Retraction Watch: https://retractionwatch.com/2020/12/09/authors-of-meta-analysis-on-heart-disease-retract-it-when-they-realize-a-nejm-reference-had-been-retracted/
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. December 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years after it was
retracted for falsifying data.” In Scientometrics, 125(3):2877–2913. doi:10.1007/s11192-020-03631-1 Preprint:
http://jodischneider.com/pubs/scientometrics2020.pdf
Schneider J, Woods ND, Proescholdt R, Fu Y, and the RISRS Team. Reducing the inadvertent spread of retracted science: Shaping a research and implementation agenda
[version 1; not peer reviewed]. F1000Research 2021, 10:211 (document) doi: 10.7490/f1000research.1118522.1

What is retraction?

Randi Proescholdt & Jodi Schneider. 2020. Retracted Papers with Inconsistent Document
Type Labeling in PubMed, Scopus, and Web of Science METRICS 2020 at ASIS&T
Many retracted articles are not indexed properly.

Appendix: Scope of the problem

How big is retraction & citation to retraction?
• 800,000 articles directly cite a retracted paper.
(Rough estimate from Fu & Schneider 2020)
• The Retraction Watch Database lists over 19,000 retracted
papers. PubMed: nearly 8,000.
• In biomedicine 94% of retracted papers have received at least
one citation, with an average citation count of 35 (Dinh, …, Schneider 2019)
Dinh, …, Schneider 2019: “Systematic examination of Pre- and Post-Retraction Citations.” doi:10.1002/pra2.35
Fu & Schneider 2020: “Towards knowledge maintenance in scholarly digital libraries with keystone citations.”
doi:10.1145/3383583.3398514

Continued citation of bad science and what we can do about it--2021-04-20

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Continued citation of bad science and what we can do about it--2021-04-20

Similar to Continued citation of bad science and what we can do about it--2021-04-20 (20)

More from jodischneider

More from jodischneider (20)

Recently uploaded

Recently uploaded (20)

Continued citation of bad science and what we can do about it--2021-04-20