Opening Keynote Presentation: The Future of Library Resource Discovery Marshall Breeding, Independent Library Consultant

1. Working out the Future of
Library Resource
Discovery
Marshall Breeding
Independent Consultant,
Founder and Publisher,
Library Technology Guides
http://www.librarytechnology.org/
http://twitter.com/mbreeding
October 5, 2015 NISO Event: Future of Library Resource Discovery
http://www.niso.org/publications/white_papers/discovery

2. Description
Marshall Breeding will highlight some of the key
findings of the white paper he developed for the
NISO Discovery to Delivery topic committee. The
presentation will include some updated information
on the state of the current arena of commercial and
open source discovery services, including trends in
adoption and new technical and functional
capabilities. Looking forward, Breeding will mention
some longer-term possibilities and opportunities for
discovery services to move beyond the current
models of centralized indexes, including greater
reliance on semantic technologies and linked data.

3. Library Discovery Past

4. Bound Catalog
National Library of Colombia

5. Card Catalog
National Library of Argentina

8. Online Catalog
 Books, Journals,
and Media at the
Title Level
 Not in scope:
 Articles
 Book Chapters
 Digital objects
Scope of Search
Search:
Search Results
ILS Data

9. NOTIS: MDAS
 Multiple Database Access System
 Released in 1989
 Article-level indexing (Mostly Wilson
Databases)
 Grant supported by Pew Charitable Trusts
 Development Partners: NOTIS and Vanderbilt
University
See: Steffey, RJ. “NOTIS multiple database access
system: a look behind the scenes” Online , v14 n5 p46-49
Sep 1990

10. Next-gen Catalogs or Discovery
Interface
 Single search box
 Query tools
 Did you mean
 Type-ahead
 Relevance ranked results
 Faceted navigation
 Enhanced visual displays
 Cover art
 Summaries, reviews,
 Recommendation services
 Books, Journals, and
Media at the Title
Level
 Other local and open
access content
 Not in scope:
 Articles
 Book Chapters
 Digital objects
 Scope of Search

11. Discovery Interface search model
Search:
Digital
Collections
ProQuest
EBSCOhost
…
MLA
Bibliography
ABC-CLIO
Search Results
Real-time query and
responses
ILS Data
Local
Index
MetaSearch
Engine

12. Web-scale Index-based Discovery
Search:
Digital
Collections
Web Site
Content
Institutional
Repositorie
s
…
E-Journals
Reference
Sources
Search Results
Pre-built harvesting
and indexing
ConsolidatedIndex
ILS Data
Aggregated
Content
packages
(2009- present)
Usage-
generated
Data
Customer
Profile
Open
Access

13. Evaluating the Performance of
Index-based Discovery Services
 Intense competition: how well the index covers
the body of scholarly content stands as a key
differentiator
 Difficult to evaluate based on numbers of items
indexed alone.
 Important to ascertain how your library’s
content packages are represented by the
discovery service.
 Important to know what items are indexed by
citation, which are full text, and how A&I
content is handled

14. Open Discovery Initiative
Libraries
Publishers
Service Providers
14
Marshall Breeding, Vanderbilt University
Jamene Brooks-Kieffer, Kansas State University
Laura Morse, Harvard University
Ken Varnum, University of Michigan
Sara Brownmiller, University of Oregon
Lucy Harrison, College Center for Library
Automation (D2D liaison/observer)
Michele Newberry
Lettie Conrad, SAGE Publications
Roger Schonfeld, ITHAKA/JSTOR/Portico
Jeff Lang, Thomson Reuters
Linda Beebe, American Psychological Assoc
Aaron Wood, Alexander Street Press
Jenny Walker, Ex Libris Group
John Law, Serials Solutions
Michael Gorrell, EBSCO Information Services
David Lindahl, University of Rochester (XC)
Jeff Penka, OCLC (D2D liaison/observer)

15. The Context for ODI
 Based on a meeting at ALA Annual Conference in
New Orleans on Sunday, June 26, 2011.
Recognition of the following trends and issues:
 Emergence of Library Discovery Services solutions
 Based on index of a wide range of content
 Commercial and open access
 Primary journal literature, e-books, and more
 Adopted by thousands of libraries around the world, and
impact millions of users
 Agreements between content providers and discovery
providers ad-hoc, not representative of all content, and
opaque to customers.
15

16. ODI deliverables
 Standard vocabulary
 NISO Recommended Practice:
 Data format & transfer
 Communicating content rights
 Levels of indexing, content availability
 Linking to content
 Usage statistics
 Evaluate compliance
 Inform and Promote Adoption
16

17. ODI Recommended Practices
 Published June 25, 2014
 NISO RP-19-2014
 http://www.niso.org/workrooms/odi/
 http://www.niso.org/workrooms/odi/publications/rp/rp-19-2014
 Metadata elements for content providers to contribute to
discovery service providers
 Content providers disclose extent to which they participate
with each discovery service
 Discovery Service providers disclose what content is
represented in index
 Discovery services disclose any bias in search results or
relevancy relative to business relationships
 Discovery services provide use statistics

18. NISO Discovery White Paper
 Commissioned by NISO Discovery to Delivery
Topic Committee
 First Draft Nov 2014
 Revised based on feedback from D2D
 Published Feb 20, 2015
 Launched at ER&L

19. NISO Discovery Paper Outline
 General Background
 Integration between Discovery Services and
Management Systems
 Linked Data
 Gap Analysis
 Opportunities for Future Enhancements in
discovery
 Discovery Beyond Library-provided Interfaces
 Open Discovery Initiative: recommendations for
Phase II
 Longer term prospects

20. Library Discovery Present

21. Library Perspective
 Strategic investments in subscriptions
 Strategic investments in Discovery Solutions to
provide access to their collections
 Expect comprehensive representation of
resources in discovery indexes
 Problem with access to resources not represented in
index
 Encourage all publishers to participate and to lower
thresholds of technical involvement and clarify the
business rules associated with involvement
 Need to be able to evaluate the coverage and
performance of competing index-based discovery
products

22. Value and Economy
 Academic and research libraries spend far more
of their budgets on content than resource
management or discovery technologies
 Discovery represents essential infrastructure to
maximize impact of library collections
 Resource management represents essential
infrastructure to assemble and assess optimal
collection to support library mission
 Ever increasing costs of content exert pressure on
budgets and demand more effective discovery
and more efficient management

23. Role of the library in discovery
 Acquisition and Management of resources
 Integrate content into campus enterprise
infrastructure and information architecture
 Provide general and specialized interfaces
 Participate in production and publication
 Participate more deeply in research process
 Manage content on behalf on the institution in
ways that optimize access and discovery.

24. Web-scale Index-based Discovery
Search:
Digital
Collections
Web Site
Content
Institutional
Repositorie
s
…
E-Journals
Reference
Sources
Search Results
Pre-built harvesting
and indexing
ConsolidatedIndex
ILS Data
Aggregated
Content
packages
(2009- present)
Usage-
generated
Data
Customer
Profile
Open
Access

25. Bento Box Discovery Model
Search:
Digital
Collections
Web Site
Content
Institutional
Repositorie
s
E-Journals
Search Results
Pre-built harvesting
and indexingConsolidatedIndex
ILS Data
Aggregated
Content
packages
Open
Access
VuFind /
Blacklight

26. State of Discovery indexes
 Very strong coverage of primary publishers of
scholarly materials
 Especially English and other Western Languages
 Weaker coverage of scholarly content in other
international regions
 Asian languages, Arabic, etc.
 Mixed coverage of A&I resources
 Mixed converge of non-textual resources

27. Some Key Areas for Publishers
1. Expose content appropriately
2. Trust that access to material will be
controlled consistent with subscription
terms
3. “Fair” Linking
4. Materials not disadvantaged or
underrepresented in library discovery
implementations
5. Usage reporting

28. Representation of A&I
 Important to understand how a discovery
service incorporates A&I resources
 Does it receive content from the A&I provider
directly and make use of value-added terminology
 If not: citations or full-text indexing of some
portion of the titles represented in the A&I product
 NOT the same, and possibly misleading
28

29. A&I Content in Discovery
Services
 What is the place for A&I services in the
discovery ecosystem
 Are there technology solutions capable of
substituting for A&I content?
 Specialized and scoped search methodologies
 Clustering, term extraction, etc.?
 Specialized vocabulary and other metadata
make positive contributions to the discovery
process
 Researchers value A&I tools

30. ODI Standing Committee
Libraries
Publishers
Service Providers
30
Marshall Breeding, Independent Consultant
Laura Morse, Harvard University
Jason Price, SCELC
Ken Varnum, University of Michigan
Dave Whisenant, Florida Virtual Campus
Lettie Conrad, SAGE Publications
Michael McFarland, CredoReference
Jill O’Neill, NFAIS
Elise Sassone, Springer
Aaron Wood, Ingram Content Group
Julie Zhu, IEEE
Scott Bernier, EBSCO Information Services
Steven Guttman, Proquest
Rachel Kessler, Ex LIbris
John McCullough, OCLC

31. ODI Standing Committee
The Open Discovery Initiative Standing Committee was formed
following approval of the Recommended Practice published by NISO
on June 25, 2014
We are charged with the following tasks:
• Promotion and education of ODI Recommended Practice for all
stakeholders
• Provide support for content providers and discovery service providers
during adoption and completion of conformance checklists
• Provide a forum for ongoing discussion related to all aspects of
discovery platforms for all stakeholders
• Consider next steps for items deemed out scope from the original ODI
Work Group Recommended Practice
• Identify emerging needs in the open discovery space and determine
appropriate courses of action
• Make recommendations to the D2D topic committee on further work
items required to fulfill the goals of the Open Discovery Initiative
31

32. Current issues and areas of
development

33. Challenge for Relevancy
 Technically feasible to index hundreds of
millions or billions of records through Lucene
or SOLR
 Difficult to order records in ways that make
sense
 Expectation that relevancy be neutral relative
to content source or publisher
 Many fairly equivalent candidates returned for
any given query
 Must rely on use-based and social factors to
improve relevancy rankings

34. Relevancy
 Ever-improving, yet flaws remain
 Increased use of use data and personalize
context to identify and order search results
 State of the art improving via more
sophisticated search and retrieval technology,
increased use of aggregated contextualized
data, and other factors

35. Socially-powered discovery
 Leverage use data to increase effectiveness of
discovery
 Usage data can identify important or popular
materials to inform relevancy engines
 Identify related materials that may not
otherwise be uncovered through keyword
matching
 Be careful to avoid introducing bias loops

36. Externalizing functionality
 Provide tools and widgets in course
management platforms
 Reading list management
 Improving presentation via mobile devices

37. Open access content
 Only a minority of scholarly resources
available through open access licenses
 Difficult to identify open access versions
available
 Often presented proprietary content when
open access is also available

38. Interoperability of Discovery
Services and Management
Platforms
 Discovery and Management solutions offered as matched
sets
 Ex Libris: Primo / Alma
 ProQuest: Summon / Intota
 OCLC: WorldCat Discovery Service / WorldShare Platform
 Independent Discovery and Management
 Kuali OLE: no discovery component
 EBSCO Discovery Service: Works with any Resource
management system
 Both product categories depend on an ecosystem of
interrelated knowledge bases
 API’s exposed to mix and match, but are efficiencies and
synergies are lost?
 Recommendation to explore expectation regarding
interoperability between these two product categories

39. Discovery Service Installations
Product 2007 2008 2009 2010 2011 2012 2013 2014 Installed
EBSCO EDS 1774 2634 8246
Primo 12 37 53 506 111 101 98 88 1528
AquaBrowser 55 339 64 69 74 58 81 6 89
Encore 72 72 109 56 72 36 346
BiblioCommons 41 ~200
Summon 50 164 214 158 238 195 697
Enterprise 16 75 100 102 123 150 538
Infor Iguana 18 74
Axiell Arena 61 57 33 35 95 404

40. Gap Analysis
 Many resources still not addressed in central
indexes
 Especially A&I products
 Better coverage of open access materials
 Better support for internationalization and
multilingual search and retrieval
 Improved capabilities for precise search, known
items, browsing
 Improved and more transparent relevancy
rankings
 Non-textual content and retrieval mechanisms
 Better integration with learning management

41. Opportunities for Enhancements in
Discovery
 Improved delivery of APIs
 More coherent ecosystem of APIs among
discovery services and with resource
management systems
 Social features and scholarly collaboration
 Address research data
 Special Collections and archival materials:
hierarchical discovery and browsing
 Expanded Analytics and Altmetrics

42. Beyond Index-based Discovery
Library Discovery Future

43. The future of Resource
Discovery
 More comprehensive discovery indexes
 Stronger technologies for search and retrieval
 Discovery beyond library-provided interfaces
 Linked Data to supplement discovery indexes

44. Universal participation
 Barriers to participation soften as mutual
interest prevails over competitive conditions
 Advantage to content providers to maximize
exposure of resources
 Discovery providers gain value in functionality
as metadata becomes increasingly
commoditized
 Essential to preserve value of indexing and
abstracting services
 Content providers see discovery as a essential
channel for distribution

45. More Distributed Discovery
 Address the reality that discovery takes place
outside of library provided interfaces
 Optimized exposure in the ecosystem of
search engine and social network
 Not Concentrated on the Library web site
 Expression of discovery services via other
campus tools and portals and beyond

46. Multi-layered discovery
 Native interfaces of specialized content
services
 Disciplinary aggregations
 General library discovery tools
 Global Internet-based discovery

47. Discovery beyond Library
Interfaces
 Improved performance of library content
through Google Scholar
 Same expectations for transparency?
 Better exposure of library-oriented content
 Schema.org or other microdata formats
 Better exposure of scholarly resources
 Open access & Proprietary
 Embedded tools in other campus interfaces

48. Part of the General Internet
Infrastructure
 Scholarly content will be promoted via similar
mechanisms as commercial content
 Additional levels of infrastructure to protect
privacy
 Resource management and/or discovery tools
expose content items as open linked data

49. Library opts out of Discovery
 Utrecht University Library
 Decision to not implement a discovery service but
to rely entirely on Google Scholar and other
general and scholarly search engines
 http://www.uu.nl/en/university-library/searching-
for-literature/searching-for-articles-books-theses
Kortekaas , Simone. “Thinking the unthinkable: a library without
a catalogue — Reconsidering the future of discovery tools for
Utrecht University library.” LIBER General Annual Conference
2012

50. Linked Data
 Major trend toward information systems based on
linked data
 Many projects now based on linked data
 Area of peak interest for Library of Congress, OCLC,
etc
 BIBFRAME
 Potential to transform how libraries approach
discovery
 Likely interim hybrid models: central indexes +
Linked Data
 Current opportunities in making library content
more discoverable

51. Library adoption of Linked data
architecture
 Not yet a fully operational method for library-
oriented content
 Increasing representation of bibliographic
resources
 BIBFRAME stands to make great impact
 Universe of scholarly resources not well
represented
 Will current expectations for content providers
to make metadata or full text available for
discovery expand to exposure as open linked
data?

52. Hybrid models
 Can index-based search tools be improved
through Linked Data
 Browse to related resources
 Add additional hierarchies of structure to search
results

53. Will linked data models prevail?
 Possibility that open linked data may
eventually supplant index-based products?
 Index technology supplements fundamental
architecture based on linked data

54. Possibilities for Open Access
discovery index
 Open source tools exist for discovery
Interfaces:
 VuFind
 Blacklight
 No open access discovery indexes
 High threshold of expense and difficulty to build
index
 Platform costs
 Software development
 Publisher relations
 Billions of content items to index and maintain

55. Current model requires massive
resources
 Threshold of resources required currently too
high for open access central discovery index
 Assessment might change if options narrowed
 Opportunities to lower barriers to entry?
 More open model more likely to come through
linked data discovery model

56. Commoditization of Central
Indexes
 Knowledgebases of e-resource coverage
commoditized via KBART and other factors
 Central index content likewise will eventually
become commoditized
 Limited number of discovery service
platforms?
 Value found in the synergies between library
resource management and optimized
discovery and delivery

57. Value in open scholarship
 Hopefully the future will be based on open
access to scholarly research
 Mandates from funding organizations will
transform scholarly communications
 Current discovery models based on
preponderance of proprietary content
 Future discovery must assume dominance of
open access publishing and underlying data
sets

58. Future of discovery service
products
 Remain one of the essential components of library
technology infrastructure
 Loosely or tightly tied to resource management
 Increased sophistication in direct discovery and
delivery functionality
 Increased expectation to syndicate content to
local and global discovery context
 Investments made in creation of discovery service
platforms will provide leverage into each next
phase of scholarly information infrastructure
 Scholarly publishing arena may change
dramatically in next decade.

59. Open Discovery Initiative:
recommendations for Phase II
 Address A&I concerns to improve participation
 Data exchange mechanisms: metadata +
content
 Lower threshold of participation
 Interoperability with resource management
systems

60. Potential Opportunities for
NISO
 Convene a second phase of the Open Discovery
Initiative
 Launch research project on open linked data in
scholarly publishing sector to facilitate new
models of discovery and access
 Expand scope of Altmetrics group to address their
integration in discovery service ecosystem
 Possible new workgroup to explore recommended
practices for improving discoverability of
resources via open linked data, schema.org, and
other mechanisms.

61. Longer term prospects
 Opportunities for discovery directly tied to
realities in scholarly publishing
 Dominance of proprietary publishing requires
index-based discovery
 Future to open access and exposure as open
linked data will enable additional models of
discovery

62. An ongoing conversation
 Now in a critical point for discovery
 Current products evolve
 Reaching limits of the prevailing architecture?
 Current set of products and services an interim
step
 Important for stakeholders to engage in
defining the future of library resource discovery
 Future products must address expected
changes in scholarly publishing, library
priorities, and institutional strategies.

63. Questions and discussion