Semantic structuring and linking of event-centric data in the social web

WP4 - semantic structuring and
linking of event-centric data
in the social web
Raphaël Troncy
raphael.troncy@eurecom.fr / @rtroncy

WP4: Original Workplan
Task Title Duration
T4.1 Specification of an event model for representing
personal events
M1 – M32
T4.2 Linking personal events to multimedia content from
the wider world
M1 – M32
T4.3 Linking personal events to knowledge and promoting
diverse and opinionated interaction
M1 – M32
T4.4 Linking with people and discovering new contacts M1 – M32
T4.5 Promoting sustainable interaction between users M1 – M32
04/06/2013 - ALIAS Final Meeting - Munich, Germany - 2
EURECOM: 76 PM
TUM-MMK: 4 PM

What are Events?
Events are observable occurrences grouping
04/06/2013 - ALIAS Final Meeting - Munich, Germany
Experiences documented by Media
People Places Time
- 3

http://www.flickr.com/photos/crsan/3697785107

User Tasks and Challenges
ALIAS Final Meeting - Munich, Germany
Goals:
 Discover PAST, PRESENT and FUTURE events
 Live, relive and predict experiences through shared media
 Enhance search and recommendations mechanisms
 Support the decision making process
Challenges
 Semantization of Data (events, media, people)
 Reconciliation of sparse and highly heterogeneous data
 User-centric design: discovering, deciding and recommending
 Scalability, Caching and Optimization
04/06/2013 - - 5

Scenario on e-tickets
 Supporting the elderly in finding events
and purchasing tickets online
 Enabling the elderly to re-live past events they have
attended with their family and relatives
 Need data about:
 Events: http://linkedevents.org/ontology/ (LODE)
 Media: http://www.w3.org/TR/mediaont-10/
 Tickets: http://www.heppnetz.de/ontologies/tio/ns.html (TIO)
 Meteo: http://purl.org/ns/meteo#
 Personalization
Local event guides are more appropriate!

Events through the web …
Event Directories
Media Platforms
Social Networks
Allows opportunistic/serendipitous discovery
Limited social features and Information
incompleteness (low coverage, weak decision)
Aids remembering and sharing past experiences
Lack of consistent events descriptions
Attendance: popularity and friends' whereabouts,
attendees’ opinions.
Limited discovery mechanism: need to filter out
relevant information.
Explore semantic connections to create better overviews of events
- 7

There’s a lot of information out there…

A lot of information…
http://www.flickr.com/photos/mwparenteau/432039783

Approach
ALIAS Final Meeting - Munich, Germany04/06/2013 -
 Aggregate heterogeneous data sources
 Unify into one data model using Semantic
Web technologies
 Enrich with media, social data and
background knowledge
 Identify meaningful and/or interesting
relationships between events, media, and
people
- 10

Architecture (generic dashboard)
Real-time Reconciliation
- Tag-based mapping
- Instance matching
Heterogeneous data sources
Data Crawling
RDF Conversion
RSS Update
Web Applications
SPARQL
REST API (Elda)
04/06/2013 - - 11

WP4.1: Data (Event) Model
 Ontology for Events:
LODE: http://linkedevents.org/ontology/
 Extension for personal events:
Based on WP1 outputs (YOU, TUM-GSING, PME)
Taxonomy of event categories
 Triple store gathering:
Events and media descriptions: user’s long-term
memories
System interaction: user’s short-term memory
- 12

Data Modeling – LODE ontology
04/06/2013 -
LODE is a minimal model that encapsulates the factual properties of
events: What, Where, When and Who.
- 13

Róisín Murphy at Nouveau Casino
350591
E0-001-005971169-9
- 14

Representing Events with

Róisín Murphy at Nouveau Casino

Media explicitly associated with the event
2.3+ million images
over 120.000+ events
Machine tags
“lastfm:events”
APIs
11.000+ videos
over 3000+ events
- 17

Representing Media with Media Ontology

Data Crawling
 Various API specifications
 Policy management
 Requests chaining
 Different input and output schemas
 Time-consuming
A Restful Service leveraging the commonalities of API specifications
A Web dashboard to easily handle data collection and interlinking
http://eventmedia.eurecom.fr/dashboard/
04/06/2013 - - 19

Real-Time Reconciliation
Participants
Media
Cost & description
Precise time
agents
04/06/2013 -
Need for semantic connections between events, media, agents and locations
- 20

Live Update
Tag = *:event
Per week
• 1500 Photos
• 130 events
Number of Items posted per days of the weekNumber of Items posted during the last days
04/06/2013 - - 21

Interlinking
Event
Media
MusicBrainz
DBpedia
Geonames
Foursquare
Uberblic
Freebase
Agent
 Last.fm
 Eventful
 MusicBrainz
 DBpedia
 Freebase
 Uberblic
 New York Times
Lieu
 Last.fm
 Eventful
 Upcoming
 DBpedia
 Freebase
 Foursquare
 Geonames
Évènement
 Last.fm
 Eventful
 Upcoming
 DBpedia
 Freebase
 Uberblic
- 22

Instance Matching
 Events/Agents/Locations semantic connections
SPARQL-based Instance matching
Filtering by the issuance date
 Challenges
Structural heterogeneity
Different representations of the same value
User-generated knowledge featuring human mistakes
Different granularities between data sources

 The events similarity is a mutual agreement of their factual properties
 Based on top-k dependencies between properties
 Evaluation: Precision: 96% and Recall: 94%
 Ground truth of 300 matched instances
Events connections
p1 p2
dependency
title1 title2 0.30
place1 place2 0.28
title1 agent2 0.26
agent1 agent2 0.21
description1 title2 0.16
Minimal conditions to fetch
similar events using SPARQL
1st level
Refine the results
2nd level
04/06/2013 - - 24

 Agents connections (precision-oriented)
 Internal reconciliation + DBpedia, Musicbrainz and BBC
 Label matching + contextual filtering (Cosine)
 Granularity? (1 to N mappings)  Cosine inequality
 Locations connections (precision-oriented)
 Internal reconciliation + DBpedia and Foursquare
 It is straightforward thanks to the consistency of locations descriptions
 Top-k Google search
 Geographical distance
Agents & Locations connections
Cosine (d, a+b) > Cosine (d ,a)
04/06/2013 - - 25

About the dataset
 SPARQL endpoint: http://eventmedia.eurecom.fr/sparql
 Restful API : http://eventmedia.eurecom.fr/rest/event/
Event Agent Location Photos User
Last.fm 61,173 62,577 15,917 1;464,885 130,249
Upcoming 19, 276 15 10,269 308,148 3,932
Eventful 45,204 8,725 18,763 3,028 321
Total 125,650 71,313 44,948 1,760,787 159,263
DBpedia MusicBrainz Eventful
Last.Fm 13064 34247 2865
Eventful 2515 3616 -
04/06/2013 - - 26

WP4.2: Linking Events with Media
 Start with media assets explicitly associated
with some events
 Enrich the dataset using large scale multimedia
analysis combined with event properties
(time, location, topic)
Learning, training, re-training: image and speech!
Apply to personal media collections and enrich gradually

User Uploading Behavior Analysis
 Most of event-related photos are uploaded in
five days.

Achievements
 Devise a generic approach for collecting media
originating from specific events based on their
visual characteristics, capture date, location
and owner.
 Propose two alternative approaches for
assessing visual similarity between canditate
media and known event media:
Visual matching against known event media (k-NN)
Visual event model trained from examples (SVN)

The Framework (Visual Matching)
 The medias are collected via the Flickr API,
thanks to the events’ multiple facets
 Visual Pruning and Owner Refinement are
employed to improve the performance
location
time
title
id
Event
query by
machine tag
query
by geo
query
by text
Media
Owner
Refine
Results
Pruning
Media
Media
- 30

Media Collection
• Multimodal features are exploited for
collecting data:
• Location
• Time
• Text
Flickr
API
- 31

Visual Pruning and Owner Refinement


Event Enricher results

Automated Event Modeling Framework
Event
tags
Rank tags
by frequency
tag1
tag2
tag3
……….tagN
Rank Photos
by distance to tags
Top N
tags
Pic1
Pic2
Pic3
PicM ……
Positive
Samples
Negative
Samples
Top M
Photos
Event
Model
- 35

Results
EventID Query Our Algorithm k-NN Pruning
lastfm:804783 87.92 88.68 46.98
lastfm:1830095 74.81 78.38 80.26
lastfm:1858887 61.84 63.41 63.56
lastfm:1499065 9.47 90.53 89.94
lastfm:1787326 0.00 98.40 92.65
lastfm:1351984 96.32 96.32 55.32
lastfm:1842684 87.28 87.93 67.86
lastfm:2020655 99.21 91.80 71.69
lastfm:1301748 93.53 93.53 73.73
lastfm:1370837 83.73 85.15 73.83
ACMMM10 88.56 91.04 87.56
SIGIR2010 0.00 60.19 42.28
ACMMM07 25.42 57.62 46.61
NICECarnival2011 22.30 76.58 59.10
Average Accuracy 69.41 83.31 68.64

WP4.3: Linking Events with Knowledge
 Add semantic categories to media assets
 Re-rank and re-organize media content for
personalized user groups
Language detection
 Enrich multimedia presentation with rich
knowledge and content from the Semantic Web
Opinions: articles from newspapers (e.g. New York Times)
Topical diversity

Sharing photos vs event categories

Sharing photos vs number of participants

Sharing photos vs country of participants

WP4.4: People, Social Network and Events
 User attendance, user profiles
Popularity and friends’ whereabouts
Similar interests, network, recommendations (behaviors)
 Social networks
EventMedia Facebook application
https://www.facebook.com/TheEventMedia
(like us!)

Co-participating events in San Francisco

Towards user profiles (LDA, T = 10)

Facebook native event application

Facebook native event application
 Event model is weak:
missing rich categories, the performers and artists, etc.
the user cannot reuse existing events
 Task: design a Facebook application that
allows for creating and sharing events, and
closely interacting with EventMedia
 Task: crawl public events and related data
hosted by Facebook using the Graph API in
order to continuously feed EventMedia

EventMedia Facebook application
https://www.facebook.com/TheEventMedia
 EventMedia meets here.com
 Results (worldwide)
 4763 venues have been interlinked
 30011 events organized in those venues
 465175 photos shared on social networks
during events at those venues
 Examples:
 http://here.com/services/place/528u173z
-672bb28e202e4032848cd6a1fe4b3851
(560 events, 9462 photos) on
eventmedia
 http://here.com/services/place/040u29fp-
70d8361e99d64a4d978f107eaa69b15a
(2 events, 542 photos) on eventmedia

WP4.5: Promoting Sustainable Interaction
 Developing end-user interfaces for searching
and browsing media content
Allow opportunistic/serendipitous discovery
Allow social feature, notification
 Support decision making upon attending
events
 Module for cross lingual chat or other IM
Google translation service

Web Application

Web Application
 Challenges:
Modern single-page app on top of semantic technologies
Fast, user-friendly and responsive UI
Dealing with high variability of data
 Technologies:
Backbone.js: MVC framework enables complex and
responsive UI
Elda Linked Data API maps SPARQL queries
to RESTful URLs

What's on this evening?
ALIAS Final Meeting - Munich, Germany04/06/2013 - - 51

What's on this evening?
ALIAS Final Meeting - Munich, Germany04/06/2013 - - 52

Demo: Live your event
http://eventmedia.eurecom.fr
- 53

Demo: Parallel Facetted Browsing
http://eventmap-ui.appspot.com/
- 54

EventMedia has been awarded
http://www.youtube.com/watch?v=8dVD0SqCB6s

ALIAS Chat Module (with translation)

WP4: Deliverables (all delivered)
Deliverable Title Due date
D4.1 Module for representing events related to user’s long term and
short term memory
M7 (Jan 2011)
D4.2 Module for cross-media linking of personal events to web content,
v1
M14 (Sep 2011)
D4.3 Module for knowledge enrichment of event descriptions M14 (Sep 2011)
D4.4 Module for personalized discovery of new contacts on line M30 (Dec 2012)
D4.5 Module for retrieval of opinionated content M33 (Mar 2013)
D4.6 Module for cross-media linking of personal events to web content,
v2
M32 (Feb 2013)
D4.7 Module for recommendation of topics and activities suitable for
maintaining social contacts
M32 (Feb 2013)
D4.8 Module for cross-lingual chat support M35
(May 2013)
- 57

WP4 Publications and Impact
 26 high quality publications + 2 awards
 + 3 more under review: RecSys 2013 + Journal of Web Semantics +
Semantic Web Journal
 Semantic Web Challenge 2012 at ISWC 2012: 1st price
 IESD Challenge at EKAW 2012: 1st price
 MediaEval Benchmark
 2011: 7 participants (18 interested)
 2012: 5 participants (25 interested)
 2013: ongoing
 2 PhD Thesis:
 Xueliang Liu (03/12/2012): Event-based Social Media Data Mining
 Houda Khrouf (scheduled for the end of 2013)

Publications 1/4
1. Raphaël Troncy, Ryan Shaw and Lynda Hardman. LODE: une ontologie pour représenter des
événements dans le web de données (on HAL). In 21th Journées Francophones d'Ingénierie des
Connaissances (IC'10), pages 69-80, Nîmes, France, June 8-11, 2010.
2. Raphaël Troncy, Ryan Shaw and Lynda Hardman. LODE: une ontologie pour représenter des
événements dans le web de données (on HAL). In 21th Journées Francophones d'Ingénierie des
Connaissances (IC'10), pages 69-80, Nîmes, France, June 8-11, 2010.
3. Raphaël Troncy, Bartosz Malocha and André Fialho. Linking Events with Media. In the Open
Track of the Linked Data Triplification Challenge, colocated with the 6th International
Conference on Semantic Systems (I-SEMANTICS'10), Graz, Austria, September 1-3, 2010.
4. Raphaël Troncy, André Fialho, Lynda Hardman and Carsten Saathoff. Experiencing Events
through User-Generated Media. In (ISWC'10) International Workshop on Consuming Linked Data
(COLD'10), CEUR Proceedings Vol. 665, Shanghai, China, November 7-11, 2010.
5. Xueliang Liu, Raphaël Troncy and Benoît Huet. Finding Media Illustrating Events. In 1st ACM
International Conference on Multimedia Retrieval (ICMR'11), Trento, Italy, April 17-20, 2011
6. Houda Khrouf and Raphaël Troncy. Réconcilier les événements dans le web de données. In
22nd Journées Francophones d'Ingénierie des Connaissances (IC'11), Chambéry, France, May
16-20, 2011.
7. Martha Larson, Adam Rae, Claire-Helene Demarty, Christoph Kofler, lorian Metze, Raphaël
Troncy, Vasileios Mezaris, Gareth J.F. Jones. Working Notes Proceedings of the Multimedia
Benchmark Workshop (MediaEval 2011), CEUR Proceedings Vol. 807, editors, Santa Croce in
Fossabanda, Pisa, Italy, September 1-2, 2011.

Publications 2/4
8. Symeon Papadopoulos, Raphaël Troncy, Vasileios Mezaris, Benoît Huet and Ioannis
Kompatsiaris. Social Event Detection at MediaEval 2011: Challenges, Dataset and Evaluation. In
MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR Proceedings Vol. 807, Pisa,
Italy, September 1-2, 2011.
9. Xueliang Liu, Benoît Huet and Raphaël Troncy. EURECOM @ MediaEval 2011 Social Event
Detection Task. In MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR
Proceedings Vol. 807, Pisa, Italy, September 1-2, 2011.
10. Marieke van Erp, Willem Robert van Hage, Laura Hollink, Anthony Jameson and Raphaël
Troncy. Proceedings of the International Workshop on Detection, Representation, and
Exploitation of Events in the Semantic Web (DeRiVE'11), CEUR Proceedings Vol. 779, editors,
Bonn, Germany, October 23, 2011.
11. Houda Khrouf and Raphaël Troncy. EventMedia Live: Reconciliating Events Descriptions in the
Web of Data. In (ISWC'11) 6th International Workshop on Ontology Matching (OM'11), Bonn,
Germany, October 23-27, 2011.
12. Houda Khrouf and Raphaël Troncy. EventMedia: Visualizing Events and Associated Media.
Demo Session at the 10th International Semantic Web Conference (ISWC'2011), Bonn, Germany,
October 23-27, 2011.
13. Xueliang Liu, Raphaël Troncy and Benoît Huet. Using Social Media to Identify Events. In (ACM
Multimedia) 3rd Workshop on Social Media (WSM'11), Scottsdale, Arizona, USA, November 28-
December 1st, 2011.

Publications 3/4
14. Liu, Xueliang; Huet, Benoit. Social event discovery by topic inference. In 13th International
Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS 2012), 23-25 May
2012, Dublin City University, Ireland
15. Houda Khrouf, Ghislain Atemezing, Thomas Steiner, Giuseppe Rizzo and Raphaël Troncy.
Confomaton: A conference enhancer with social media from the cloud. In 9th Extended
Semantic Web Conference (ESWC 2012), May 27-31, 2012, Heraklion, Crete
16. Houda Khrouf, Ghislain Atemezing, Giuseppe Rizzo, Raphaël Troncy and Thomas Steiner.
Aggregating social media for enhancing conference experience. In 1st International Workshop
on Real-Time Analysis and Mining of Social Streams (RAMSS 2012), June 4, 2012, Dublin,
Ireland
17. Symeon Papadopoulos, Emmanouil Schinas, Vasileios Mezaris, Raphaël Troncy and Ioannis
Kompatsiaris. Social Event Detection at MediaEval 2012: Challenges, Dataset and Evaluation. In
MediaEval Benchmarking Initiative for Multimedia Evaluation, CEUR Proceedings Vol. 927, Pisa,
Italy, October 4-5, 2012.
18. Sven Buschbeck, Anthony Jameson, Raphaël Troncy, Houda Khrouf, Osma Suominen and
Adrian Spirescu. A Demonstrator for Parallel Faceted Browsing. In (EKAW) Intelligent
Exploration of Semantic Data Workshop (IESD 2012), Winner of the IESD Challenge, Galway,
Ireland, October 8-12, 2012.
19. Marieke van Erp, Laura Hollink, Willem Robert van Hage, Raphaël Troncy and David A. Shamma.
Proceedings of the Second International Workshop on Detection, Representation, and
Exploitation of Events in the Semantic Web (DeRiVE 2012), CEUR Proceedings Vol. 902, editors,
Boston, USA, November 12, 2012.

Publications 4/4
20. Houda Khrouf, Vuk Milicic and Raphaël Troncy. EventMedia Live: Exploring Events Connections
in Real-Time to Enhance Content. In 11th International Semantic Web Conference (ISWC 2012),
First Prize Winner of the Semantic Web Challenge, Boston, USA, November 11-15, 2012.
21. Xueliang Liu and Benoit Huet. Gathering training sample automatically for social event visual
modeling. ACM International Workshop on Socially Aware Multimedia (SAM 2012), in
conjunction with ACM Multimedia 2012, 29 October 2012, Nara, Japan
22. Symeon Papadopoulos, Emmanouil Schinas, Vasileios Mezaris, Raphaël Troncy and Ioannis
Kompatsiaris. The 2012 Social Event Detection Dataset. In ACM Multimedia Systems (MMSys
2013), Dataset Session, Oslo, Norway, February 27-March 1, 2013
23. Sven Buschbeck, Raphaël Troncy, Anthony Jameson, Houda Khrouf, Adrian Spirescu, Osma
Suominen, Tanja Schneeberger and Eero Hyvönen. Parallel Faceted Browsing. In ACM
Conference on Human Factors in Computing Systems (CHI 2013), Interactivity Track, Paris,
France, April 27-May 2, 2013
24. Xueliang Liu and Benoit Huet. EventEnricher: a novel way to collect media illustrating events.
ICMR 2013, ACM International Conference on Multimedia Retrieval, April 16-20, 2013, Dallas,
Texas, USA
25. Xueliang Liu and Benoit Huet. Heterogeneous features and model selection for event-based
media classification. ICMR 2013, ACM International Conference on Multimedia Retrieval, April
16-20, 2013, Dallas, Texas, USA
26. Xueliang and Benoit Huet. On the automatic online collection of training data for visual event
modeling. Multimedia Tools and Applications, February 2013, ISSN: 1380-7501

http://www.slideshare.net/troncy

Semantic structuring and linking of event-centric data in the social web

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

Similar to Semantic structuring and linking of event-centric data in the social web

Similar to Semantic structuring and linking of event-centric data in the social web (20)

More from Raphael Troncy

More from Raphael Troncy (20)

Recently uploaded

Recently uploaded (20)

Semantic structuring and linking of event-centric data in the social web