Presentation for the Bibliometrics & Research Assessment Symposium 2020 (bibSymp20) https://www.nihlibrary.nih.gov/services/bibliometrics/bibSymp20 October 9, 2020 Retraction is intended to remove articles from the citable literature. However, a series of studies from over 30 years, from 1990 through 2020, have found that many retracted papers continue to be cited, and cited positively, even following misconduct-related retractions. For instance, a fraudulent clinical trial report retracted in 2008 continues to receive citations in 2020, and 96% of post-retraction citations do not mention its citation - perhaps because its retraction not marked on the publisher website and its retraction notice cannot be readily retrieved from 7 out of 8 databases (8 out of 9 database records) we tested. This talk draws an ongoing systematic mapping study of research about retraction and our own research projects to summarize what is known about post-retraction citation in biomedicine. We outline practical steps that authors and reviewers can take to avoid being caught out by poorly marked retracted papers. 20 minutes including Q&A

1. The problems of post-retraction citation -
and mitigation strategies that work
Jodi Schneider
School of Information Sciences
University of Illinois at Urbana-Champaign
jodi@illinois.edu
Twitter:@jschneider
Bibliometrics & Research Assessment Symposium 2020, Online, 2020-10-09
Organized by SLA Maryland Chapter & NIH Library

2. Disclosures
• I received industry funding to present at CrossRef LIVE19
• NIH Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine
• I’m organizing a fall 2020 workshop with Sloan Foundation funding
– Retraction Watch’s Ivan Oransky is on the advisory board
– Attendees include the previous speaker (scite’s Josh Nicholson)

3. What is retraction?
“Retraction is a mechanism for correcting the literature and
alerting readers to articles that contain such seriously flawed or
erroneous content or data that their findings and conclusions
cannot be relied upon.”
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019
https://doi.org/10.24318/cope.2019.1.4

4. What is retraction?
“Retraction is a mechanism for correcting the literature and
alerting readers to articles that contain such seriously flawed or
erroneous content or data that their findings and conclusions
cannot be relied upon.”
“Prompt retraction should minimise the number of researchers
who cite the erroneous work, act on its findings, or draw
incorrect conclusions such as from ‘double counting’ redundant
publications in meta-analyses or similar instances."
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019
https://doi.org/10.24318/cope.2019.1.4

5. Problem 1: Retracted papers get cited and used

6. How big is retraction & citation to retraction?
• 800,000 articles directly cite a retracted paper.
(Rough estimate from Fu & Schneider 2020)
• The Retraction Watch Database lists over 19,000 retracted
papers. PubMed: nearly 8,000.
• In biomedicine 94% of retracted papers have received at least
one citation, with an average citation count of 35 (Dinh, …, Schneider 2019)
Dinh, …, Schneider 2019: “Systematic examination of Pre- and Post-Retraction Citations.” doi:10.1002/pra2.35
Fu & Schneider 2020: “Towards knowledge maintenance in scholarly digital libraries with keystone citations.”
doi:10.1145/3383583.3398514

7. Use of retracted papers is also problematic

8. Problem 2: Retraction status is often poorly marked

9. This RCT paper is retracted…. But how would you know?

10. Retracted October 2008 because an author falsified data

11. Difficult to get to the retraction notice via database search
• We tested 8 databases: Only 1 (EMBASE) had a working link to the retraction notice!
• Linking errors give the impression that the retraction notice doesn’t exist!
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics. In
press.

12. Problem 3: Authors cite retracted papers without
knowing they are retracted

13. Since its 2008 retraction, this RCT paper has been cited over 100 times.
Only 4% (5/112) mention the retraction.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years after it was
retracted for falsifying data.” In Scientometrics. In press.

14. Since its 2008 retraction, this RCT paper has been cited over 100 times.
Only 4% (5/112) mention the retraction.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years after it was
retracted for falsifying data.” In Scientometrics. In press.

15. Since its 2008 retraction, this RCT paper has been cited over 100 times.
Only 4% (5/112) mention the retraction.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years after it was
retracted for falsifying data.” In Scientometrics. In press.

16. Retraction doesn’t stop citation.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years
after it was retracted for falsifying data.” In Scientometrics. In press.

17. Retraction doesn’t stop citation.
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years
after it was retracted for falsifying data.” In Scientometrics. In press.
5 out of 112 post-retraction citations
mention the retraction.
The remaining 107 use the paper –
and its faked data – as normal science.

18. Problem 4: Misinformation from retracted papers
can continue to spread to further generations of
research.

19. Retraction doesn’t stop citation.
Only 5 mention the retraction
2008
32 direct citations
63 indirect published 2008
2014 2019
114 direct citations
256 indirect published 2014
148 direct citations
108 indirect published 2019
Only 2 mention the retractionRCT Retracted:
Author falsified data
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report, eleven years
after it was retracted for falsifying data.” In Scientometrics. In press.
07
Paper (black)
n articles (blue): 22
on articles (red): +25
2008
Matsuyama Paper (black)
# of first−generation articles (blue): 32
# of second−generation articles (red): +63

20. Mitigation strategy 1: Be aware of retraction

21. RetractionWatch
https://retractionwatch.com
• Retraction news
• Top 10 most cited retracted papers
• Retracted COVID-19 papers

22. Office of Research Integrity:
https://ori.hhs.gov
• Misconduct case summaries
• Research Misconduct Findings

23. Mitigation strategy 2: Search Databases

24. Check retraction status in PubMed
“Authors are responsible for
checking that none of the
references cite retracted articles
except in the context of referring
to the retraction.”
“For articles published in journals
indexed in MEDLINE, the ICMJE
considers PubMed the
authoritative source for
information about retractions.”

25. PubMed’s “retracted publication” [PT]

26. PubMed’s “retracted publication” [PT]

27. PubMed Publication Types
• Published erratum
• Expression of concern
• Corrected and republished article
• Retracted publication
• Retraction of publication
• Scientific integrity review

28. Anne O’Tate
http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/AnneOTate.cgi

29. RetractionWatch Database
http://retractiondatabase.org/

30. Many databases indicate retraction status.
• Ask vendors for better indexing!
• Display varies.
• Check multiple sources.
• Report indexing errors!

31. Mitigation strategy 4: Zotero

32. Zotero flags retracted items based on DOI
Price, Gary. 2019 June 12. InfoDocket. Zotero and Retraction Watch Collaborate on New Service (Beta) That Notifies Users of
Article Retractions in Their Personal Zotero Libraries. https://www.infodocket.com/2019/06/12/zotero-and-retraction-watch-
collaborate-on-new-service-beta-that-notifies-users-of-article-retractions-in-their-personal-zotero-libraries/

33. Zotero gives an alert about retracted items

34. Zotero adds a “Retracted Items” list

35. Future mitigation idea: Flag problematic citations

36. Step 4
Flag those articles
that are potentially
impacted
Workflow for Flagging Problematic Citations
Step 2
The domain expert
develops a list of screening
questions
Step 3
Experts/non-experts/text
mining tools screen target
articles using the checklist
Step 1
The domain expert develops
a generalized argument
model
Fu, Yuanxi, Jodi Schneider. 2020. “Towards knowledge maintenance in scholarly digital libraries with
keystone citations.” In JCDL 2020, 217–226. doi:10.1145/3383583.3398514

37. Reducing the Inadvertent Spread of Retracted Science
2020-2021
Jodi Schneider jodi@illinois.edu
Information Quality Lab https://infoqualitylab.org/
Collaboration across diverse stakeholders: funders, editors, peer reviewers, authors,
publishers, database providers, research integrity officers, science journalists
Advisory Board
National Center
for Professional
and Research Ethics
Retraction Watch
JAMA
Annettte Flanagin
RN, MA, FAAN
Ivan Oransky, MDC.K. Gunsalus, JD
London School of
Economics and Political
Science
Daniele Fanelli, PhD

38. Bibliography
Cheng, Yi-Yun, Nikolaus Parulian, Tzu-Kun Hsiao, Ly Dinh, Janina Sarol, Jodi Schneider. 2019. “ReTracker: actively and automatically
matching retraction metadata in Zotero.” In ASIS&T Annual Meeting, 56(1): 372-376. doi:10.1002/pra2.32
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019 doi:10.24318/cope.2019.1.4
Dinh, Ly, M. Janina Sarol, Yi-Yun Cheng, Tzu-Kun Hsiao, Nikolaus Parulian, Jodi Schneider. “Systematic examination of Pre- and Post-
Retraction Citations.” In ASIS&T Annual Meeting, 56(1): 390-394. doi:10.1002/pra2.35
Fu, Yuanxi, Jodi Schneider. 2020. “Towards knowledge maintenance in scholarly digital libraries with keystone citations.” In JCDL 2020, 217–
226. doi:10.1145/3383583.3398514
Harris, Richard. 2020 March 26. “In Defense Of Coronavirus Testing Strategy, Administration Cited Retracted Study.” NPR.
https://www.npr.org/sections/health-shots/2020/03/26/822084429/in-defense-of-coronavirus-testing-strategy-administration-cited-
retracted-study
Price, Gary. 2019 June 12. InfoDocket. Zotero and Retraction Watch Collaborate on New Service (Beta) That Notifies Users of Article
Retractions in Their Personal Zotero Libraries. https://www.infodocket.com/2019/06/12/zotero-and-retraction-watch-collaborate-on-
new-service-beta-that-notifies-users-of-article-retractions-in-their-personal-zotero-libraries/
Proescholdt, Randi & Jodi Schneider. 2020. Retracted Papers with Inconsistent Document Type Labeling in PubMed, Scopus, and Web of
Science METRICS 2020 at ASIS&T. In press.
Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda workshop:
https://infoqualitylab.org/projects/risrs2020/
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a fraudulent clinical trial report,
eleven years after it was retracted for falsifying data.” In Scientometrics. In press. doi (pending): 10.1007/s11192-020-03631-1
Preprint: http://jodischneider.com/pubs/scientometrics2020.pdf

39. Appendix: COPE on retraction

40. What is retraction?
COPE Council. COPE Guidelines: Retraction Guidelines. Version 2 November 2019
https://doi.org/10.24318/cope.2019.1.4

41. Appendix: Flagging important citations
with the keystone citation framework

42. Citing Article Cited Article
Cite
Support arguments
Under my keystone citation
framework:
1) A scientific research paper puts forward at
least one main finding, along with a
logical argument, giving reasons and
evidence to support the main finding.
2) The main finding is accepted (or not) on
the basis of the logical argument.
3) Evidence from earlier literature may be
incorporated into the argument by citing a
paper and presenting it as support, using a
citation context.
Citation Contexts
Arguments
modeled
Fu & Schneider, JCDL 2020, Towards Knowledge Maintenance in Scientific
Digital Libraries with the Keystone Framework

43. Appendix: Spread of misinformation to a second
generation

44. Described the Matsuyama paper’s methods and/or results (but not the retraction):
35 direct citations 2010-2019 (blue squares)
Cited by 161 articles (red squares)
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics. In press.
Specific citation 2010−2019

45. “Health benefits of flaxseed” (Fitzpatrick, 2011)
Government research bulletin from Nepal (Jha, 2016)
Spread of misinformation to a second generation
which cites the retracted paper as “evidence”
of the anti-inflammatory impact of flax.
“Whilst it is true that very little ALA
converts to the long chain
polyunsaturated omega-3 found in
marine oils, it does have beneficial
effects itself (Fitzpatrick, 2011).”
cites a book chapter
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics. In press.

46. An Irish nutritional support shop
recommends n-3 fats to athletes
(Healthy fats, fish oils & omega-
3 supplementation, 2017)
Spread of misinformation to a second generation
which cites the retracted paper as having demonstrated “Improved 6 min walk test,
decreased leukotriene B4 level, TNF-alpha, IL-8 [91]” (i.e., the faked data for which the
Matsuyama paper was retracted)
“During periods of illness, this may help promote
recovery and faster return to training. Interestingly, n-3
fats are sometimes provided to COPD patients (severe
airway damage and breathing difficulties) and prior to
surgery in order to support the immune system and speed
recovery by helping to control inflammation and
infection, and repair damaged cells17.”
“Immunologic impact of nutrient depletion in chronic obstructive pulmonary disease”,
(Herzog & Cunningham-Rundles, 2011)
cites an article
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics. In press.

47. A pre-clinical study on lung
repair following dust exposure
(Nordgren et al., 2018)
Spread of misinformation to a second generation
NO “large body of evidence” is found in this paper: the retracted paper is its only
“evidence” on n-3 PUFAs alone. The only related evidence it provides is from a review
article and an exercise combotrial. The only cited “evidence” for the effect of n-3 PUFAs on
inflammation in COPD came from the retracted Matsuyama paper.
“Furthermore, studies reveal diets high in omega-3
polyunsaturated fatty acids (n-3 PUFA) may be
beneficial in inflammatory lung conditions, including
asthma and COPD (17).”
“Beneficial effects of n-3 PUFA on chronic airway inflammatory diseases” (Giudetti
and Cagnazzo, 2012)
cites an article
Schneider, Jodi, Di Ye, Alison M Hill, Ashley S Whitehorn. 2020. “Continued post-retraction citation of a
fraudulent clinical trial report, eleven years after it was retracted for falsifying data.” In Scientometrics. In press.

48. Appendix: Data Quality

49. Randi Proescholdt & Jodi Schneider. 2020. Retracted Papers with Inconsistent Document
Type Labeling in PubMed, Scopus, and Web of Science METRICS 2020 at ASIS&T
Many retracted articles are not indexed properly.