Presented by Electra Enslow, Suzanne Fricke, Susan Shipman The use of supplemental journal article materials is increasing in all disciplines. These materials may be datasets, source code, tables/figures, multimedia or other materials that previously went unpublished, were attached as appendices, or were included within the body of the work. Current emphasis on critical appraisal and reproducibility demands that researchers have access to the complete shared life cycle in order to fully evaluate research. As more libraries become dependent on secondary aggregators and interlibrary loan, we questioned if access to these materials is equitable and sustainable.

1. NASIG 35th ANNUAL CONFERENCE | JUNE 9-11, 2020
#NASIG2020

2. Many thanks to our sponsors for making the
NASIG 35th Annual Conference possible!

3. Access to Supplemental
Journal Article Materials
NASIG 2020

4. Suzanne Fricke
Washington State University - Pullman
Animal Health Sciences Librarian
Electra Enslow
Washington State University – Spokane
Head of Library Research and Instruction
Sue Shipman
Washington State University – Pullman
Access Services Manager

5. Overview
Introduction
• Definition SJAM
• Motivation for study
• Best Practices
• Literature Review
Method
Results
• Quantitative Data
• Qualitative Data
Future

6. Introduction:
Definition of
Supplemental
Materials to
Journal Articles
(SJAM) =
materials found
outside the
article
• Tables and figures
• Checklists
• Reporting guideline flow
diagrams
• Study protocol
• Data collection/extraction
forms
• Interview transcripts
• Open laboratory notebooks
• Raw study data
• Control data
• Software codes
• Data visualizations
• Dynamic models
• Media files
• Methods
• Reagents
• Conflict of interest
statements
• Continuing education
• PICO worksheets and search
strategies for guidelines
• Sketches
• Translations
• Forms and surveys

7. Introduction:
Motivation for
study is
equitable and
sustainable
access
1. Faculty tell us they often do not receive SJAM with
interlibrary loan (ILL) requests, even when they
specifically ask for it
2. Librarians frequently ask for SJAM on librarian listservs
3. We observed through reference questions that:
• Journal access to SJAM is inconsistent across
journals and years of publication
• Links to SJAM are often broken
• Secondary aggregator versions of journal articles
vary in SJAM access from publisher versions
• New direct to PDF browser extensions (LibKey
Nomad) are not linking to SJAM

8. Introduction:
NISO/NFAIS RP-15-
2013 Recommended
Practices forOnline
Supplemental Journal
Article Materials
These recommended practices are intended to help the
scholarly publishing community develop a more standardized
approach to publishing what has become known as
Supplemental Materials for Journal Articles[SJAM] in a rapidly
changing technology environment. The intent is to lessen the
burden on all of the parties engaged in the publishing
process, to ensure that Supplemental Materials delivered in
connection with journal articles add substance to the
scholarship, to make Supplemental Materials more
discoverable, and to aid in preserving them.
• Publishers should provide written guidelines for acceptance
or outsource archiving of SJAM
• SJAM should be reviewed at the same level as the article
• SJAM should have a persistent identifier
• SJAM should have standardized naming conventions,
metadata including file types, indexing and navigation in
the table of contents, article and the SJAM
• Treatment of bibliographic references based on SJAM
designation as integral or additional

9. Introduction: Literature Review
• Use of SJAM has increased (Kenyon & Sprague, 2014; Schaffer & Jackson, 2004) – due to electronic
publishing, article length restrictions, demand for reproducibility and need for critical appraisal
• Studies have mainly looked at disciplinary publishing practices (Evangelou et al., 2005; Kenyon &
Sprague, 2014; Schaffer & Jackson, 2004; Schriger,Chehrazi, Merchant, & Altman, 2011; Shutler &
Murray, 2016;Vidal-Infer et al., 2019;Williams, 2016) or surveyed authors/reviewers(Price et al.,
2018)
• Journals lack consistent policies – one file, separate files, tied to elements of the article, no SJAM,
disciplinary repository with identifier
• SJAM may not be adequately reviewed (Cheifet, 2018;Pop & Salzberg, 2015)
• Current standards do not stipulate organization and file types (Cheifet, 2018)
• Most share as PDF orWord document (Kenyon, 2016; Pop & Salzberg, 2015; Maunsell, 2010; Alter &
Gonzalez, 2018;Price et al, 2018;Greenbaum et al., 2017)
• Growing need for machine readable data (JimenoYepes, 2014)

10. Research Statement
This study aims to understand librarians and
library staff members experience with SJAM,
and provide recommendations for identifying,
finding and retrieving SJAM.

11. Method:
Survey librarians and library
staff members
• Qualtrics survey design
• Demographics – library type and size, amount and
frequency of experience with ILL and SJAM
• Open-ended – experience with SJAM, patron
assumptions and issues with SJAM
• IRB
• Survey distribution
• academic, special and medical librarians
• Library ILL and access services staff
• Quantitative and qualitative analysis

12. Quantitative
Data:
Demographics
Completion of Survey (total listserv distribution 21,451)
• Total opened survey (1%) 211
• Total Answered US Question 197
• Total in US 191
• Total moved past US question 180
• Total moved past library size question 120
LibraryType (N=180)
• Hospital 56 29%
• Academic 107 56%
• Public 0 0%
• Other 17 9%
• Government
• Research institute
• Health education center
• Professional organization
• Publisher
Library Size range 50-200,00

13. Quantitative
Data:
Completion of
survey by
library type

14. Quantitative
Data:
Work with ILL
by library type

15. Quantitative
Data:
SJAM
Demographics
How often asked for SJAM (N=191)
• Never 5 3%
• 1-5 x/year 39 20%
• 6-10 x/year 25 13%
• More than 10x/year 37 19%
• Other (Always, Don’t know) 13 7%
• No response 72 38%
Institution provides SJAM when filling ILL requests (N=191)
• Yes, always look and provide if accessible 44 23%
• No, only provide if requested 56 29%
• Other (unsure, if aware, if easy, Docline only) 16 8%
• No response 75 39%

16. Quantitative
Data:
At what point is
patron aware
of SJAM

17. Quantitative
Data:
SJAM requests
vs work with
ILL

18. Quantitative
Data:
Work with ILL
vs provide
SJAM when not
requested

19. Quantitative
Data:
Patrons
assume SJAM
included vs
provide
without asking

20. Qualitative
Data:
Open ended responses coded using
grounded theory to reveal
themes/subthemes

21. Qualitative
Data:
Theme One
Why Supplem
ental Materials
are a problem
Behind publisher pay wall
“We have struggled with getting supplementary materials for [publisher] articles requested through ILL.The
supplementary materials are often accessible through a link that as a non-subscriber, we (our patron) cannot access.”
Separate from article
“I wish the publishers would include them with the original article as an appendix instead of separate document.”
Multiple files
“I would like it if articles with multiple supplements would put them all in one file. If an article has 8 separate
supplements in 8 separate files, I won’t retrieve and send for ILL unless requested. If all 8 supps are in one file, I will.”
Multiple file types
“Often if there is supplemental material it is only visible thru the HTML view, and not the PDF.This presents a problem as
the lender attaches PDF files not HTML, so the reader discovers the references to supplemental material after receiving
the main article in PDF.
Not Indexed
“We also struggle with doi not reflecting a difference between the original article and SJAM”
“a consistent format for listing available SJAM with the article would improve everyone’s awareness”
ILL Systems do not support supplemental materials
“Part of the problem with obtaining SJAMs is that ILLiad is not really equipped to handle them . . . which is something
that needs to be addressed soon by the vendor, in my opinion.”
Not available through secondary aggregators
“Packages we buy through [secondary aggregators] never identify or deliver supps.”

22. Qualitative
Data:
ThemeTwo
Lack of
Understanding
of Supplemental
Material
By Patrons
“Most of our patrons do not know about supplementary materials when requesting
articles. But are a few heavy users that always ask for supplementary materials.”
“I as the librarian catch it and attach the supplements.”
By Librarians and Library Staff
“I hadn’t heard of them. Don’t e-articles include all related things like charts, tables,
etc.?”
“when we ask other libraries for the SJAM, they provide another copy of the article
and not the SJAMs. It’s very frustrating.”
By publishers
“I find that patrons have difficulty finding SJAM even for articles that we do have,
particularly since seems as if every publisher has a different layout for identifying
SJAMs”
“It seems that SJAMs are an afterthought for journal editors and publishers.When
you edit a manuscript, you forget that the author(s) put the survey in an appendix or
offered additional data – so it dangles.”

23. Qualitative
Data:
ThemeThree
Expectations for
Supplemental
Material
Patrons expect supplemental material
“Some can be quite indignant that the SJAM weren’t supplied along with
the article”
“They would not mention it to me until they get the article without the
supplement. Some do indicate on the initial request that that they want
the supplement, but others just seem to expect it.”
Requesting libraries expect that lending libraries are looking for
supplemental material
“I attach several (1-25) SJAMs to Lend almost every day, but rarely ever
receive them for my Borrow I.L.L.s even though I specifically ask for
them.”
“WHY can’t they just make a download that INCLUDES all the
supplemental material (in addition to the regular article download). That
way, other libraries could just click that and send it, instead of my arguing
with them about sending it. I actually don’t have that problem as much
anymore because I deleted the problem lenders from our ILL routing
table. I don’t want the stress!”

24. Qualitative
Data:
Theme Four
Barriers to
Routinely
Supplying
Supplemental
Material
Time
“Time consuming and cumbersome; that’s why we have stopped
supplying unless specifically asked for.”
“Publishers really need to make the content more obvious.We
can’t skim every article to see if they mention supplemental
material.”
Considered something special
“depends on how serious they are about the article”
Not in the habit
“getting veteran ILL technicians to get in the habit of including
suppl mat is another hurdle”
Difficult
“It’s a pain in the neck to download all of those supplementary files in
various formats. Often there are quite a number of files!”

25. Future
Recommendations

26. Future Recommendations
Publishers,Aggregators, Databases, and Repositories
• Comply with all NISO recommended practices for SJAM
• Provide SJAM in one file or as a permanent link with doi
• Index SJAM consistently in a way that facilitates discovery of SJAM without reading the
article
• Fix broken links to SJAM as stated in NISO best practices
• Assure that the industry develops a collective understanding of data as a stand-
alone item, increasingly not intended to be linked with one study
• Publishers, aggregators, databases, repositories and scholarly search engines
should use consistent/standard terminology, organization, file formats, and
systems of reporting SJAM access problems

27. Future
Recommendations
Library Systems
• Library ILL systems should have a way to
communicate to the patron that the library
checked for supplemental materials, and a
way to designate if it was unavailable
• Create ILL systems that are not PDF
dependent and can accommodate the
variety of file types made available by
publishers (HTML, etc)

28. Future Recommendations
Librarians and Library Staff
• Understand how important SJAM is to critical appraisal
• Train academic library staff to identify the presence of SJAM using database indexing
and supply SJAM even when not requested
• Adopt a standard practice of supplying supplemental materials
• Communicate effectively with patrons about library policies on providing SJAM and
instruct patrons on how to best access SJAM
• Understand that data is increasingly separate for a reason. Data is no longer associated
with just one article. It is an entity in and of itself

29. Future Recommendations
Patrons and Liaison
Librarians
• Learn to use database indexing to identify the
presence of supplemental materials
• Learn to always look for and request SJAM

30. Impact of Open Access and now COVID-19
• Improved indexing in open access repositories
• PubMed Central has a requirement that all supplemental materials be supplied with articles submitted to
PMC, even when they are available in a public repository, and designated with unique identifiers and
associated XML
• Pre-print servers - MedRxiv and BioRxiv consistently identify the presence of supplementary materials and
data/code
• No change
• Lack of indexing in most library databases and scholarly search engines (Google Scholar, PubMed) – this
requires further study
• Lack of standard terminology – example: "associated data" in PubMed vs Web of Science
• Lack of file format standards - Word files in MedRxiv

31. Future Recommendations
Best Practices/Guidelines/Standards
• Provide guidance for consistent terminology, access to expanding file formats and
linked data
• Discontinue the distinction between integral and additional content as that can only be
determined by the reader
• Provide short plain language summaries and infographic visualization of standards for
training of technical staff

32. References
Alter, G., & Gonzalez, R. (2018). Responsible practices for data sharing. Am Psychol, 73(2), 146-156. doi:10.1037/amp0000258
Cheifet, B. (2018). Making supplemental information more accessible. Genome Biology, 19(1), 225. doi:10.1186/s13059-018-1609-8
Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19(14), 1943-1944.
Flanagin, A., Christiansen, S. L., Borden, C., Kyriacou, D. N., Sietmann, C., Williams, E., & Bryant, L. (2018). Editorial Evaluation, Peer Review, and Publication of Research Reports With and Without
Supplementary Online Content. JAMA, 319(4), 410. doi:2670244 [pii] 10.1001/jama.2017.20650 [doi]
Greenbaum, D., Rozowsky, J., Stodden, V., & Gerstein, M. (2017). Structuring supplemental materials in support of reproducibility. Genome Biology, 18(1), 64. doi:10.1186/s13059-017-1205-3
Jimeno Yepes, A., & Verspoor, K. (2014). Literature mining of genetic variants for curation: quantifying the importance of supplementary material. Database (Oxford), 2014, bau003. doi:bau003
[pii]10.1093/database/bau003 [doi]
Kenyon, J., Sprague, N., & Flathers, E. (2016). The Journal Article as a Means to Share Data: a Content Analysis of Supplementary Materials from Two Disciplines. Journal of Librarianship & Scholarly
Communication,4, 1-17. doi:10.7710/2162-3309.2112
Kenyon, J., & Sprague, N. R. (2014). Trends in the Use of Supplementary Materials in Environmental Science Journals. Issues in Science & Technology Librarianship(75), 7-7. doi:10.5062/F40Z717Z
Maunsell, J. (2010). Announcement Regarding Supplemental Material. The Journal of Neuroscience, 30(32), 10599-10600.
Pop, M., & Salzberg, S. L. (2015). Use and mis-use of supplementary material in science publications. BMC Bioinformatics, 16, 237. doi:10.1186/s12859-015-0668-z
Price, A., Schroter, S., Clarke, M., & McAneney, H. (2018). Role of supplementary material in biomedical journal articles: surveys of authors, reviewers and readers. BMJ Open, 8(9), e021753. doi:10.1136/bmjopen-
2018-021753
Schaffer, T., & Jackson, K. M. (2004). The Use of Online Supplementary Material in High-Impact Scientific Journals. Science & technology libraries., 25(1-2), 73-85. doi:10.1300/J122v25n01_06
Schriger, D. L., Chehrazi, A. C., Merchant, R. M., & Altman, D. G. (2011). Use of the Internet by print medical journals in 2003 to 2009: a longitudinal observational study. Ann Emerg Med, 57(2), 153-160.e153.
doi:10.1016/j.annemergmed.2010.10.008
Shutler, D., & Murray, A. (2016). Trends, costs, benefits, challenges, and prognoses for supplementary materials. The Auk: Ornithological Advances, 133(4), 733-737.
Vidal-Infer, A., Aleixandre-Benavent, R., Lucas-Dominguez, R., & Sixto-Costoya, A. (2019). The availability of raw data in substance abuse scientific journals. Journal of Substance Use, 24(1), 36-40.
doi:10.1080/14659891.2018.1489905
Williams, S. C. (2016). Practices, Policies, and Persistence: A Study of Supplementary Materials in Crop Science Journals. Journal of Agricultural & Food Information, 17(1), 11-22.
doi:10.1080/10496505.2015.1120213

33. Questions?
suzanne.fricke@wsu.edu
electra.enslow@wsu.edu