Europeana cloud jun 2016
•Download as PPTX, PDF•
1 like•775 views
Presentation about the lessons learned from the project and how we plan to move forward with the design of shared services for data partners.
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
More from Europeana
More from Europeana (9)
Recently uploaded
Recently uploaded (20)
Europeana cloud jun 2016
- 2. Title here Subtitle here Europeana Cloud Review CC-BY-SA
- 3. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 4. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 5. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 6. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 7. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 8. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 9. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 10. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 11. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 12. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 13. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 14. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 15. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 16. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 17. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 18. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 19. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 20. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 21. Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA Title here Subtitle here Business Plan 2016 CC-BY-SA Europeana Cloud Review CC-BY-SA
- 23. Images denkschets.nl CC BY-SA
Editor's Notes
- We conceived the project nearly 5 years ago with a set of great expectations about the use of cloud computing in our sector. It had the promise of providing shared processing resources and data to computers and other devices on demand. More efficiently, cheaper. A term as old as the 1990’s, the idea became big after 2006 with the introduction of Amazon’s Elastic Cloud services. But by the time we started the project in february 2011 still very new to the heritage sector, and we had yet to come to terms with it- quite literally. This has been quite a challenging project for us. A complex and large infrastructural innovation project with 33 partners is always difficult to manage. Our understanding and vision have changed over the course of the past 3 years. …
- <intermediate slide to outline the story>
- So let’s take a step back and see what we set out to achieve first: We want to transform the world with culture, and Europeana’s platform strategy is based on the premise that we need to make it incredibly easy and rewarding for cultural institutions to share their data through Europeana and more widely, on the web.
- The Europeana Cloud project (started first of February 2012, conceived mid 2011, 33 partners) set out to do exactly that: make it easier for aggregators to store, manage and share data using cloud technologies. It had 7 defined objectives, which including building a working storage system incorporating 2.4 million new metadata records and 5 million content items for an intended audience of researchers in the digital humanities. While the project aimed at improving all aspects of data management, the emphasis of the project was on creating a storing infrastructure for aggregators. This was based on hypothesis that cost of storage was a major pain-point for aggregators (and down the line for cultural institutions). Cloud technology promised to cut up to 20% of these costs.
- Besides the hard facts the project was also underpinned by a large set of less well defined expectations. The organisational hypothesis was that we would probably spin-off a separate entity based on commons principles. This would ensure shared ownership and possibly an income stream to support the Foundation. The project also assumed that we would achieve a ‘broad consensus among European content aggregators and research networks on the advantages of a cloud based solution.’
- So let’s see what happened: The project delivered this infrastructure, which is now in use by the three project partners Europeana, TEL and PSNC.
- <project achievements>
- 1. IaaS (Infrastructure as a Service: A storage service for metadata and digital media capable of scaling up to store massive amounts of data and to/from which data can be written and retrieved over an API. This is based at the Poznan Supercomputing Centre. Basic generic data processing services were developed which include an identifier system, an authentication and notification system, logging and lookup. With the Europeana newspaper migration of 18 million pages we will surpass the intended amount of 5 million but at this point in time it is unclear how much data is stored in the system: Open University 3.2 million records, Poznan 1.8 million thumbnails, 1.7 million metadata records from TEL.
- 2. SaaS (Software as a Service): a small number of specific data processing services to demonstrate the capability of the generic data processing service, the most prominent being the IIIF service (which transforms image files into JPG2000 and makes them available over IIIF in “zoomable” form)
- 3. Europeana Research: a service aimed at making use of Europeana metadata and media easier for researchers and research infrastructures
- The world has changed in the past 3 years and so have the expectations. We have interviewed 13 large aggregators in 2015 (3 nationals, 10 domain). DDB Hispana Culturitalia EU Screen EFG Fashio Hope Apex OpenUp CARARE Michael / MUSEU Food and Drink, Sounds So what have we learned?
- 1.The first is that cost of storing data is not the primary pain point for aggregators and cultural institutions. Of course storing data needs to be solved, but there are plenty of companies offering a good service and the costs of storage are measured in $ per GB still dropping by the minute. Storage infrastructure (IaaS) is perhaps not a standalone service to market, but very useful as an enabler for specific services built on top (PaaS). 2. However, our research has shown that there seems to be a strong need in the cultural sector for value added services that everyone who wishes to publish with Europeana can benefit from. 3. The current workflow for processing data is still slow, opaque and cumbersome. We need to make intrinsically motivating for GLAMs to publish with us.
- That the need to generate income from the market has diminished in favour of developing useful tools that save costs. A separate entity is therefore probably not needed.
- So what are the next steps? 1. We will use the basic infrastructure and further develop Value added services like the IIIF (Sound? Video?)
- 2. Under the heading of Metis we will develop specific services like data validation and previewing that will make it easier for aggregators to share data with Europeana.
- 3. But we also want to take it one step further and test whether we can develop services that enable cultural institutions to publish with Europeana directly, completely without intermediary (operation direct). Europeana will aim to make these services available for free or at cost to partner institutions who will make their data available through Europeana, and who will contribute to the platform and the network. This means open metadata, open code and following the Publishing Framework etc.