Austrian Books Online. The Austrian National Library's Large-Scale Digitisation Public-Private Partnership with Google

@maxkaiser
Austrian Books Online
The Austrian National Library’s
large-scale digitisation public-private partnership
with Google
Max Kaiser
Head R&D, Austrian National Library
Library Science Talk
Geneva, 15 October 2012
Bern, 16 October 2012

@maxkaiser
www.onb.ac.at/ev/austrianbooksonline/

@maxkaiser
www.slideshare.net/maxkaiser

@maxkaiser
digitisation
of the entire historical
book holdings of the
Austrian National Library

@maxkaiser
largest Austrian
public private partnership
in the cultural sector

@maxkaiser@maxkaiser

@maxkaiser
history back to the
14th century

one of the world‘s
most significant
collections

Quelle:
http://commons.wikimedia.org/wiki/File:A
ustria_Hungary_ethnic_de.svg
„legal deposit“

@maxkaiser
legal deposit today
→print publications
→online publications
→web archiving

seven special collections

@maxkaiser
→ Picture Archives and Graphics Department
→ Map Department
→ Music Department
→ Literary Archives
→ Papyri Department
→ Department of Planned Languages
→ Department of Rare Books and Manuscripts

music department

four museums

@maxkaiser
→ State Hall
→ Papyrus Museum
→ Globe Museum
→ Esperanto Museum

papyrus department & museum

Department of Planned Languages
& Esperanto Museum

Globe Museum

16 reading rooms

9am–9pm – 7 days/week

library as social space

@maxkaiser@maxkaiserservices for researchers

@maxkaiser
access for everyone
from anywhere

+10 mio. pages
historical newspapers & legal texts

several 100k
images

140k
portraits

100k* posters
*by end 2012

papyri…

@maxkaiser
→ September 2012
http://www.onb.ac.at/
about/21043.htm

@maxkaiser
Vision 2025Knowledge for the world of tomorrow
Our holdings are digitized
We collect and sustain knowledge
Access to our knowledge is simple
With us, research is more faceted and effective
We enrich cultural and social life

@maxkaiser
→substantial parts of holdings digitized
→cooperation with private partners
→full text search
→added-value services like semantic search
→unified access system
Our holdings are digitized

@maxkaiser
→focal point of collection policy is digital
→preference for digital versions of publications
→user generated content and social networks
→digital photography
→preservation of analogue and digital
collections
→scalable digital archive
We collect and sustain knowledge

@maxkaiser
→unified access system for all collections
→focus of cataloguing: metadata enrichment
→linking of metadata with external resources
→open data
→APIs and support for third party apps
Access to our knowledge is simple

@maxkaiser
→integration of digital content in virtual
research environments
→support for digital humanities
→strong research collections and libraries
→cooperation with universities and research
centres
With us, research is more faceted and simple

@maxkaiser
→digital services, reading rooms and
museums
→innovative interfaces
→mobile services
→cooperation with private partners: reuse
of data for innovative services
→reinforce library as social space
We enrich cultural and social life

@maxkaiser

@maxkaiser
600,000 volumes
200 Mio pages

@maxkaiser
16th century
2nd half of
19th century _

@maxkaiser
Google Books
Digital Library

@maxkaiser
Partner Program
Library Program
Google Books

@maxkaiser
13 Libraries in Europe
5 National Libraries
 Italy
 Austria
 The Netherlands
 Czech Republic
 Great Britain

@maxkaiser
>20 Mio. books
> 50% non-English
~ 75% from libraries
~ 2 Mio. books from European libraries
> 3 Mio. books public domain

@maxkaiser
some strategy and policy considerations…

@maxkaiser
is a
public private
partnership?

@maxkaiser
service contract or service outsourcing
→long duration of the relationship
→substantial investment by private
partner
→distribution of risks
≠

@maxkaiser
rationales for PPPs
→private funding for Public Sector
→benefit from know-how and working
methods of the private sector
→but not a „miracle solution“
for the public sector
(EC Green Paper on Public Private Partnerships, 2004)

@maxkaiser
public private
partnerships in the
cultural sector

@maxkaiser
objectives for public partners
→funding for digitisation
→enhanced access
→engaging new audiences
→access to technology
→access to private sector competencies
→commercial income through user fees,
royalties or revenue share
→lobbying effort to increase public funding

@maxkaiser
objectives for private partners
→commercial objectives
→access to new markets or customer groups
→association with strong public brands
→access to (rare, unique) content
→corporate social responsibility

@maxkaiser
benefits for citizens
→increased online access
→democratisation of access to knowledge
→added-value services
→benefit for learning and tourism
→new creative endeavours

http://ec.europa.eu/information_society/activities/digital_libraries/
doc/reflection_group/final_report_%20cds.pdf
10 January 2011

@maxkaiser
„Stimulating the flow of private funds
for the digitisation of cultural assets through
equitable public private partnerships
appears as a viable and sustainable way
of tackling the pressing question
of making Europe’s cultural wealth
accessible online and preserving it
for future generations.“

@maxkaiser
„The key question is not
whether public-private
partnerships for digitisation
should be encouraged, but
how‚ and under which
conditions.“

@maxkaiser
„(...) recommends that Member States (...)
encourage partnerships between cultural
institutions and the private sector in
order to create new ways of funding
digitisation of cultural material and to
stimulate innovative uses of the material,
while ensuring that public private
partnerships for digitisation are fair and
balanced (…).“

key principles:
1. respect for intellectual property rights
→ ONB-Google: only public-domain works
digitised
2. non-exclusivity
→ ONB-Google: ONB free to digitise material
with other partners
3. transparency of the process
→ ONB-Google: public tender

@maxkaiser
key principles:
4. transparency of agreements
→ ONB-Google: Very detailed FAQs online
5. accessibility through Europeana
→ ONB-Google:
→ all files available for non-commercial use
→ access via platforms like Europeana
→ provision to research partners
6. key criteria
→ [Next slide]

@maxkaiser
key criteria for assessing PPPs
→ total investment by private partner / effort of
public partner
→ (free) access to material for general public,
including through Europeana
→ cross-border access
→ length of any period of preferential commercial
use by private partner
→ quality of digital copies for public partner
→ usage conditions for public partner in non-
commercial context
→ time-scale of project

@maxkaiser
additional key elements in
ONB-Google cooperation:
→selection of books by library
→Institute for Conservation involved
→termination

@maxkaiser
„Genuine PPPs currently not a widespread
method for financing digitisation by cultural
institutions in Europe.“
Commission Staff Working Paper Accompanying the document Commission Recommendation
on the digitisation and online accessibility of cultural material and digital preservation, p18
http://ec.europa.eu/information_society/activities/digital_libraries/doc/recommendation/recom28nov_all_versions/staff_working_paper.pdf

@maxkaiser
aim to maximize access
and re-use via digitisation
access restrictions /
re-Use limitations in PPPs

@maxkaiser
public private partnerships
as commodification
of the cultural commons?

@maxkaiser
Cultural Commons
→Body of work freely available to the public for
legal use, sharing, repurposing, and remixing
→Source for cultural creativity
→http://creativcommons.org/culture

@maxkaiser
Public Domain
→material to derive knowledge and create new
cultural works
→essential for society and economy

@maxkaiser
http://www.europeana-libraries.eu/web/europeana-project/publications

@maxkaiser
Public Domain Mark
„This work has been identified
as being free of known
restrictions under copyright
law, including all related and
neighbouring rights.
You can copy, modify,
distribute and perform the
work, even for commercial
purposes, all without asking
permission.“
http://creativecommons.org/publicdomain/mark/1.0/

@maxkaiser
Public Domain Charter
„Public-Private Partnerships have become one
option for funding large scale digitisation efforts.
Commercial content aggregators pay for the
digitisation in exchange for privileged access to the
digitised collections. These activities are seen as a
reason for attempting to exercise as much control as
possible over digital reproductions of Public Domain
works. Organisations are claiming exclusive rights in
digitised versions of Public Domain works and are
entering into exclusive relationships with commercial
partners that hinder free access.”

@maxkaiser@maxkaiser@maxkaiser
Public Sector Information
→information produced, collected and held by
public institutions
→single largest source of information in Europe
→should be widely re-used to foster economy
and creativity

@maxkaiser
PSI Directive
→EC “Directive on the Re-Use of Public Sector
Information” (31 Dec. 2003)
→aim: Foster re-use of PSI
→legally binding document
→implemented by all Member States in 2008
→currently: Cultural & research institutions
excluded from directive

@maxkaiser
key provisions of PSI Directive
→clear procedures for re-use requests
→upper limit for charging
→transparency of conditions and standard
charges for re-use
→avoid discrimination between players
→prohibition of exclusive agreements

@maxkaiser
→12 Dec 2011:
Commission proposal
for PSI Directive
amendment

@maxkaiser
proposed changes
→withdraw current exemption for cultural
institutions
→restrict public sector bodies to only apply
charges for re-used based on marginal
costs
→exemption for libraries, archives, museums
→prohibit agreement of terms for re-use
which grant exclusive rights to any one
party

@maxkaiser
→discussion in Council Working Groups
under Danish and Cyprus Presidencies
→latest published draft: 1 Oct. 2012

http://register.consilium.europa.eu/pdf/en/12/st13/st13162.en12.pdf

@maxkaiser
→Working Draft, 1 Oct. 2012: Article 11

who is paying
for what?
http://www.bildarchivaustria.at/downl/1148453/layout/CE%2043_3.jpg

@maxkaiser
costs
→full text-digitisation:
very expensive
→report by
Collections Trust
for Comité des Sages
http://ec.europa.eu/information_society/activities/digital_libraries/
doc/refgroup/annexes/digiti_report.pdf

@maxkaiser
Google:
→transport
→insurance
→scanning
→OCR
→image processing
→quality control
→Google Books

@maxkaiser
Austrian National Library:
→ provision of Metadata
→ selection
→ internal logistics
→ conservational assessment
→ barcoding
→ metadata adjustments
→ data download and control
→ data storage & digital preservation
→ Digital Library

→conservation
→preservation
http://www.mediathek.at/akustische-chronik/popup/popup.php?document_id=1000115&zone_id=
1000043&template_id=1000016&zone_name=IMAGE_ZONE1

entire historical
book holdings
16th –19th century

200.000 volumes
State Hall

Quelle: http://deu.archinform.net/projekte/107
Department of Manuscripts
and Rare Books
Map Department

Quelle: http://commons.wikimedia.org/wiki/File:Palais_Lobkowitz_Vienna_Oct._2006_006.jpg
Theatre Museum

@maxkaiser@maxkaiser@maxkaiser

@maxkaiser
7 Work Packages
 Book logistics
 Metadata / Catalogues
 Conservation / Restoration
 Data download / Quality control
 Access
 IT infrastructure
 Project management

@maxkaiser
preparatory project
mid - end 2010
→integration with organisational processes
→personnel resources
→logistics workflows

@maxkaiser
internal communication
→change processes
→re-evaluation of workflows
→availability of internal resources

consultation with other
Google partners
Quelle: http://commons.wikimedia.org/wiki/File:M%C3%BCnchen_Bayerische_Staatsbibliothek_001.JPG

@maxkaiser
70+ staff members
20+ exclusively for project
→ book logistics
→ metadata adaptation
→ cataloguing
→ conservation / restoration
→ quality control
→ software implementation
→ project management

@maxkaiser
end of 2010
test shipment & start operational project
Spring 2011
start of digitisation

logistics in the
„Aurum“ Depot

conservational protection

cataloguing the
Fidei Commiss Library

@maxkaiser
digitisation
→ scanning Center in Germany
→ procedures agreed
→ Austrian Federal Office for Monuments involved
→ each volume checked after return
→ books unavailable to users for ~ 3 months

@maxkaiser
where are we today?

@maxkaiser100.000volumes digitized
today

@maxkaiser185.000volumes digitized
by end 2013

@maxkaiser
of 100.000 volumes:
9,19% 16th century
14,24% 17th century
31,48% 18th century
43,01% 19th century
2,07% [no year of publication]

@maxkaiser
of 100.000 volumes:
33,41% German
31,31% Latin
15,55% French
13,78% Italian
2,73% English

@maxkaiser
book flowdigital flow

@maxkaiser
digitisation
data download
book logistics
quality control
storage
access
ADOCO
(Austrian Books Online
Download & Control)

@maxkaiser
up to
digitised items / day

@maxkaiser
quality control
→goal: Automated jobs
→representative samples
→IT assisted discovery of error clusters
→error candidates checked manually
→detect systematic
and critical errors

@maxkaiser
error model
→ level 1: data / information
→ image (thick, broken)
→ illustration (scanner effects, tone, color etc)
→ full-text (OCR errors per page-image)
→ level 2: entire page
→ blur / warp / skew
→ cropping
→ obscure / cleaned
→ colorization
→ full-text (OCR error patterns at page level)
Informed by „Validating Quality in
Large-Scale Digitization“ project
of Univ. of Michigan & Univ. of Minesota,
http://hathitrust-quality.projects.si.umich.edu/

@maxkaiser
error model
→ level 3: whole volume
→ order of pages
→ missing pages
→ duplicate pages
→ false pages
→ full text (OCR error patterns at volume level)

@maxkaiser
use cases
→reading online images
→printing on demand
→processing full text data
→managing collections

@maxkaiser
non-critical
errors

@maxkaiser
quality control
via sampling
re-processing
re-download

@maxkaiser
cropping
error
FIXED

@maxkaiserbig data processing…
http://blogs.loc.gov/digitalpreservation/files/2012/05/3875300483_a8875fea1c-500.jpg

@maxkaiser
technologies and workflows
from EC co-funded FP7 projects:
→SCAPE
(Scalable Preservation Environments)
→http://www.scape-project.eu/
→IMPACT
(Improving Access to Text)
→http://www.impact-project.eu/

hadoop / map reduce
SLAVE 1
Task Tracker
Data Node
SLAVE 2
Task Tracker
Data Node
SLAVE n
Task Tracker
Data Node
MASTER
Job Tracker
Name Node
Hadoop Distributed File System (HDFS)
→ experimental 5 server cluster at ONB:
→ 40 cores in total
→ 30 cores assigned to task trackers

@maxkaiser
use case 1: duplicate pages
in one book
→books with duplicated pages
→due to scanning process & post processing
→use key points of images to determine
structural image similarity

use case 1: duplicate pages
in one book

@maxkaiser
use case 2: book comparison
based on image similarity
→different instances of one book, coming
→e.g. from different downloads of one book
at different points in time
→book similarity measure
→based on comparison of book page images
from two different book instances

use case 2: book comparison
based on image similarity
measure for book similarity
based on book page image
similarity
 helps finding prominent
changes in book re-
downloads

@maxkaiser
large scale document processing
→extract image metadata using Exiftool
→large scale batch processing using Apache
Hadoop Streaming API
→bash script using Exiftool is executed on the
cluster
→book page image data is accessible from
each node of the cluster
→parallelisation of batch processing

find
/NAS/Z119585409/00000001.jp2
/NAS/Z119585409/00000002.jp2
/NAS/Z119585409/00000003.jp2
…
/NAS/Z117655409/00000001.jp2
/NAS/Z117655409/00000002.jp2
/NAS/Z117655409/00000003.jp2
…
/NAS/Z119585987/00000001.jp2
/NAS/Z119585987/00000002.jp2
/NAS/Z119585987/00000003.jp2
…
/NAS/Z119584539/00000001.jp2
/NAS/Z119584539/00000002.jp2
/NAS/Z119584539/00000003.jp2
…
/NAS/Z119599879/00000001.jp2l
/NAS/Z119589879/00000002.jp2
/NAS/Z119589879/00000003.jp2
...
...
NAS
reading files from NAS
1,4 GB 1,2 GB
~ 5 h + ~ 38 h = ~ 43 h
60.000 books
24 mio pages
Jp2PathCreator HadoopStreamingExiftoolRead
Z119585409/00000001 2345
Z119585409/00000002 2340
Z119585409/00000003 2543
…
Z117655409/00000001 2300
Z117655409/00000002 2300
Z117655409/00000003 2345
…
Z119585987/00000001 2300
Z119585987/00000002 2340
Z119585987/00000003 2432
…
Z119584539/00000001 5205
Z119584539/00000002 2310
Z119584539/00000003 2134
…
Z119599879/00000001 2312
Z119589879/00000002 2300
Z119589879/00000003 2300
...

@maxkaiser
large scale document processing
→ store once in HDFS and read many times
→ small files (TXT, HTML) stored in HDFS
→ files of each file type stored as one big file
(SequenceFile)
→ principle: store once in HDFS and read many times
→ example:
→ storing OCR results of 24 mio pages (ca. 60.000
books)  reading data from file server and storing on
cluster takes more than 1 day
→ subsequent processing of a Map/Reduce job (calculate
average block width) takes 6 hours

find
/NAS/Z119585409/00000707.html
/NAS/Z119585409/00000708.html
/NAS/Z119585409/00000709.html
…
/NAS/Z138682341/00000707.html
/NAS/Z138682341/00000708.html
/NAS/Z138682341/00000709.html
…
/NAS/Z178791257/00000707.html
/NAS/Z178791257/00000708.html
/NAS/Z178791257/00000709.html
…
/NAS/Z967985409/00000707.html
/NAS/Z967985409/00000708.html
/NAS/Z967985409/00000709.html
…
/NAS/Z196545409/00000707.html
/NAS/Z196545409/00000708.html
/NAS/Z196545409/00000709.html
...
Z119585409/00000707
Z119585409/00000708
Z119585409/00000709
Z119585409/00000710
Z119585409/00000711
Z119585409/00000712
NAS
reading files from NAS
1,4 GB 997 GB (uncompressed)
~ 5 h + ~ 24 h = ~ 29 h
60.000 books
24 mio pages
HtmlPathCreator SequenceFileCreator

Z119585409/00000001
Z119585409/00000002
Z119585409/00000003
Z119585409/00000004
Z119585409/00000005
...
~ 6 h
Z119585409/00000001 2100
Z119585409/00000001 2200
Z119585409/00000001 2300
Z119585409/00000001 2400
Z119585409/00000002 2100
Z119585409/00000002 2200
Z119585409/00000002 2300
Z119585409/00000002 2400
Z119585409/00000003 2100
Z119585409/00000003 2200
Z119585409/00000003 2300
Z119585409/00000003 2400
Z119585409/00000004 2100
Z119585409/00000004 2200
Z119585409/00000004 2300
Z119585409/00000004 2400
Z119585409/00000005 2100
Z119585409/00000005 2200
Z119585409/00000005 2300
Z119585409/00000005 2400
Z119585409/00000001 2250
Z119585409/00000002 2250
Z119585409/00000003 2250
Z119585409/00000004 2250
Z119585409/00000005 2250
Map/Reduce
HadoopAvBlockWidthMapReduce
SequenceFile Textfile
example map/reduce job:
calculate average block width

Sqoop
combine MySQL and Apache Hive DB
book level
metadata
page level
metadata

@maxkaiserstorage and access…

@maxkaiser
data
average size data package (~book): 101 MB
colour data package: 187 MB
grayscale data package: 82 MB
101 MB * 600.000 = 60 TB

@maxkaiser
storage & access
→ data storage: in-house
→ JPEG-2000 master files stored redundantly
→ access copies generated on-the-fly
→ URN resolver for permanent identification

@maxkaiser
book viewer
catalogue /
“Quick Search”
full-text search
[mobile apps]

Image Server
Catalogue
URN Resolver
Master Images
Digital Repository
Quick Search
Fulltext
Index Server
Book Viewer
ADOCO
Google
USER

@maxkaiser
outlook
→ full-text: new possibilities for research
→ data enrichment
→ named entity recognition
→ linked data
→ new data centric research in the Humanities
& Social Sciences
→ http://www.diggingintodata.org/

@maxkaiser
DM2E
→http://dm2e.eu/
→European Commission co-funded project
→stimulate creation of new tools and
services for re-use of Europeana data in
the Digital Humanities
→implementation of semantic annotation
tool
→Austrian Books Online data part of the
project

@maxkaiser
next steps
→80.000 books already accessible via
Google Books
→Spring 2013: launch of Austrian Books
Online Viewer
→full text search

@maxkaiser
http://books.google.at/books?vid=ONB%2BZ15367990X

@maxkaiser
http://books.google.at/books?vid=ONB%2BZ155606704

@maxkaiser

@maxkaiserhttp://books.google.at/books?vid=ONB%2BZ169472305

@maxkaiser

@maxkaiser
more information
www.onb.ac.at/ev/austrianbooksonline
www.onb.ac.at/ev/austrianbooksonline/faq.htm
twitter.com/abooksonline

@maxkaiser
thank you!
max.kaiser@onb.ac.at
www.onb.ac.at
www.slideshare.net/maxkaiser
www.linkedin.com/in/maxkaiser
gplus.to/maxkaiser
twitter.com/maxkaiser

Austrian Books Online. The Austrian National Library's Large-Scale Digitisation Public-Private Partnership with Google

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Austrian Books Online. The Austrian National Library's Large-Scale Digitisation Public-Private Partnership with Google

Similar to Austrian Books Online. The Austrian National Library's Large-Scale Digitisation Public-Private Partnership with Google (20)

More from Max Kaiser

More from Max Kaiser (15)

Recently uploaded

Recently uploaded (20)

Austrian Books Online. The Austrian National Library's Large-Scale Digitisation Public-Private Partnership with Google