Representing and creating events and lining them in EventWeb may result in powerful systems to represent dynamic situations. This is one of the next evolutionary stages of the Web.

1. EventWeb
Ramesh Jain
and
Several Collaborators
Contact: jain@ics.uci.edu
ICIG Xian Sept 21, 2009 © Ramesh Jain 1

2. Today
• Data, Experience, and Knowledge
• Semantic Gap
• Bridging the Semantic Gap
• EventWeb
• Applications:
– EMME: Experiential Media Management Environment
– E2E: Environment to Environment Connection
ICIG Xian Sept 21, 2009 © Ramesh Jain 2

3. Name 3 most popular companies that
emerged in the last 3 years.
• YouTube
• Facebook
• Twitter
ICIG Xian Sept 21, 2009 © Ramesh Jain 3

4. Twitter’s Growth
This is a growth pattern I have never seen in terms
of speed - not in the nearly 25 years I've been
watching this industry. --- John Battelle
ICIG Xian Sept 21, 2009 © Ramesh Jain 4

5. Name 3 most popular Internet
concepts in the last 3 years.
• Social Networks
• Micro-blogging (Ambient Awareness)
• Tags
News: None of these come from GYM or Academia.
ICIG Xian Sept 21, 2009 © Ramesh Jain 5

6. Interesting!!!
• YouTube • Social Networks
• Facebook • Microblogging
• Twitter • Tags
What does this tell us?
ICIG Xian Sept 21, 2009 © Ramesh Jain 6

7. Messages for Research
Community
People want:
• New media: Text based media is not enough.
• Experiences: People want to experience and
share experiences – with minimal latency.
• Socialize: Family and friends remain a strong
influence in all facets of life.
– Family and friends are closer to each other today than
ever!!!
ICIG Xian Sept 21, 2009 © Ramesh Jain 7

8. Events
Life = +
Experiences
ICIG Xian Sept 21, 2009 © Ramesh Jain 8

9. Recording experiences
• Visual
• Aural
• Tactile
• (not worry about smell and taste for the
next few years)
• Text
• Log of activities
ICIG Xian Sept 21, 2009 © Ramesh Jain 9

10. ICIG Xian Sept 21, 2009 © Ramesh Jain 10

11. ICIG Xian Sept 21, 2009 © Ramesh Jain 11

12. The Challenge
Connecting
ICIG Xian Sept 21, 2009 © Ramesh Jain 12

13. Transformations
Lists, Arrays, Documents, Images …
Alphanumeric Characters
Bits and Bytes
ICIG Xian Sept 21, 2009 © Ramesh Jain 13

14. Semantic Gap
The semantic gap is the lack of coincidence
between the information that one can
extract from the visual data and the
interpretation that the same data have for
a user in a given situation. A linguistic
description is almost always contextual,
whereas an image may live by itself.
Content-Based Image Retrieval at the End of the Early Years
Found in: IEEE Transactions on Pattern Analysis and Machine Intelligence
Arnold Smeulders , et. al., December 2000
ICIG Xian Sept 21, 2009 © Ramesh Jain 14

15. Semantic Gap
ICIG Xian Sept 21, 2009 © Ramesh Jain 15

16. History: Year 1991
– We had Internet.
– Lots of computers were connected to each
other.
– Computers had files on them.
– We had GOPHER and other FTP
mechanisms.
ICIG Xian Sept 21, 2009 © Ramesh Jain 16

17. Tim Berners-Lee thought:
• Suppose all the information stored on
computers everywhere were linked.
• Suppose I could program my computer to
create a space in which anything could be
linked to anything.
Others – including Bush -- had
that idea earlier but the
technology was not ready.
ICIG Xian Sept 21, 2009 © Ramesh Jain 17

18. That resulted in the Web
• DocumentWeb
– Each node is a ‘Page’ or a document.
– Pages are linked through explicit referential
links
ICIG Xian Sept 21, 2009 © Ramesh Jain 18

19. Today we have
• WWW that has revolutionized our life in
many ways.
• Sensor networks
– Billions of sensor getting connected
• Ambitious projects
– Planetary Skin by Cisco and NASA
– Smart Planet by IBM
ICIG Xian Sept 21, 2009 © Ramesh Jain 19

20. Can we take the Web beyond the
Cyberspace?
• Can things in real world be connected to
other things?
• Does this even make sense?
ICIG Xian Sept 21, 2009 © Ramesh Jain 20

21. Connectors
• My 5 Senses are connectors between
‘me’ and the world.
• We use our sensors (vision, audio, …) to
experience the world.
• Sensors could be the interface between
the Cyberspace and the Real World.
• Sensors are placed for ‘detecting events’.
– How do you decide what sensors to put at
any place?
– Would you put a sensor if nothing interesting
ever happens at a place?
ICIG Xian Sept 21, 2009 © Ramesh Jain 21

22. Modeling the World
• Data (Semantic Web)
• Objects (Search Companies, …)
• Relationships and Events
ICIG Xian Sept 21, 2009 © Ramesh Jain 22

23. Evolution of ‘Thought’
• Objects -- popular in the West.
• Relationships and Events – popular in the
East.
• Objects and Events – seems to be the new
trend.
• The Web has reemphasized the importance
of every object and event being connected to
others -- East Meets West.
ICIG Xian Sept 21, 2009 © Ramesh Jain 23

24. Continuing the Evolution of
the Web
• Consider a Web in which each node
– Is an event
– Has informational as well as experiential data
– Is connected to other nodes using
• Referential links
• Structural links
• Relational links
• Causal links
– Explicit links can be created by anybody
• This EventWeb is connected to other Webs.
ICIG Xian Sept 21, 2009 © Ramesh Jain 24

25. Different types and Granularities
• Conferences
– Days
• Sessions
– Talks
» Purpose of the talk
• Wedding
• An Earthquake
• The Big Bang
• 9/11
• 26/11
• Formation of Google
• Attending WWW
• Me
– My Birth,
– Being here, and
– Dying in 100 years.
ICIG Xian Sept 21, 2009 © Ramesh Jain 25

26. Event Representation
ICIG Xian Sept 21, 2009 © Ramesh Jain 26

27. Transformations
Text, Images, Audio, Video, Tactile…
Alphanumeric, Pixel, Characters
Bits and Bytes
ICIG Xian Sept 21, 2009 © Ramesh Jain 27

28. Events
1- dimensional Space
Time
ICIG Xian Sept 21, 2009 © Ramesh Jain 28

29. EventWeb
1- dimensional Space
Time
ICIG Xian Sept 21, 2009 © Ramesh Jain 29

30. Multimedia Storytelling
• Collect information about events
– Select relevant events
– For each event, select appropriate information
– In right media
• Stories are sequence of coherent events.
– Stories/Novels
– Drama
– Movies
Present right event information
using right media in right order.
ICIG Xian Sept 21, 2009 © Ramesh Jain 30

31. Multimedia Storytelling
1- dimensional Space
Text
Photo
Video
Time
ICIG Xian Sept 21, 2009 © Ramesh Jain 31

32. Stages in EventWeb
• Cyberspace: Every event is ‘created’
in cyberspace and exists there.
• Physical and Cyberspace: Events in
real space are also linked. We have
Sentient EventWeb linked with the
Cyber Web.
ICIG Xian Sept 21, 2009 © Ramesh Jain 32

33. Experiential Media Management
Environment
• Event-based
• Should be able to deal with ‘multimedia’
– Photos
– Audio
– Video
– Text
– Information and data
– …
• Searching based on events and media.
• Storytelling
First Photos then other media.
ICIG Xian Sept 21, 2009 © Ramesh Jain 33

34. Content Contenxt Context
• Contenxt = Content + Context
• Context is as powerful, possibly more, as
content in understanding audio-visual
information
ICIG Xian Sept 21, 2009 © Ramesh Jain 34

35. Modern Cameras
• Are more than ‘Camera Obscura’: They capture an
event.
• Many sensors capture scene context and store it
along with intensity values.
• EXIF data is all metadata related to the Event.
Exposure Time
Aperture Diameter
Flash
Metering Mode
ISO Ratings
Focal Length
Time
Location (soon)
Face ICIG Xian Sept 21, 2009 © Ramesh Jain 35

36. Photos can be
tagged using
only EXIF
We will also use content
features and LSCOM
Concepts – will soon start
using them.
Information from
calendars and other
sources will be introduced
soon.
ICIG Xian Sept 21, 2009 © Ramesh Jain 36

37. EMME Event Cycle
Event Atomic
Presentation/ Event Entry
Navigation
EXIF
Features
Event Base Tags/
Context
Event Grouping, Linking, 2009
ICIG Xian Sept 21, Assimilation © Ramesh Jain 37

38. Using Context/Models to
Build the EventWeb
• Folder structure
• Calendar
• Social Network
• EXIF Data
• Event Ontology
• Personal annotations
• Photostream Segmentation
• Event Detection from photos
ICIG Xian Sept 21, 2009 © Ramesh Jain 38

39. Photo Stream Segmentation
Definition: given a photo stream P = { pi }
Segment
Event 1 Event 2 Event 3
ICIG Xian Sept 21, 2009 © Ramesh Jain 39

40. EMME Event Cycle
Event Atomic
Presentation/ Event Entry
Navigation
EXIF
Features
Tags/
Event Base Context
User Photo
Annot- Event
stream
ations Ontology Segment.
Minimize Manual Work
Event Grouping, Linking, 2009
ICIG Xian Sept 21, Assimilation © Ramesh Jain 40

41. EMME Event Cycle
Event Atomic
Presentation/ Event Entry
Navigation
Story
Telling
EXIF
Explore
Features
Search Tags/
Event Base Context
User Photo
Annot- stream
ations Event Segment.
Ontology
Event Grouping, Linking, 2009
ICIG Xian Sept 21, Assimilation © Ramesh Jain 41

42. Using EMME
• Searching for photo
– ISWC2008
• Creating Albums:
– Professional
– Family
– Tourism
• Telling stories
– What did I do in Karlsruhe
• Scenario: In December 2008, I have 20,000 pictures taken in 2008.
How do I (semi-automatically) select 25 to send to
– My mother
– The uncle that I hate Version 0.1
– My personal friend is ready
– My professional friend
– …
ICIG Xian Sept 21, 2009 © Ramesh Jain 42

43. Personal-Photo-EventWeb
ICIG Xian Sept 21, 2009 © Ramesh Jain 43

44. Singapore – Outdoor --
People
ICIG Xian Sept 21, 2009 © Ramesh Jain 44

45. People-No Face - Outdoor
ICIG Xian Sept 21, 2009 © Ramesh Jain 45

46. Singapore-Outdoor-Night
ICIG Xian Sept 21, 2009 © Ramesh Jain 46

47. Indoor no faces
ICIG Xian Sept 21, 2009 © Ramesh Jain 47

48. Indoor People
ICIG Xian Sept 21, 2009 © Ramesh Jain 48

49. Indoor Portraits
ICIG Xian Sept 21, 2009 © Ramesh Jain 49

50. E-mail can also be parsed.
ICIG Xian Sept 21, 2009 © Ramesh Jain 50

51. Photos Summarization
ICIG Xian Sept 21, 2009 © Ramesh Jain 51

52. Original Data Set
ICIG Xian Sept 21, 2009 © Ramesh Jain 52

53. Photo-Summarization using content
ICIG Xian Sept 21, 2009 © Ramesh Jain 53

54. Photo-Summarization using Faces
ICIG Xian Sept 21, 2009 © Ramesh Jain 54

55. Event Mash-up Interface
Event Mash-up Interface
Composite Events
Event Digestor
Event Digestor
Clustering Annotation Event Ontology
Atomic Events
Event Ingestor
Event Ingestor
EXIF Extractor Text Parser Calendar Parser
ICIG Xian Sept 21, 2009 © Ramesh Jain 55

56. ICIG Xian Sept 21, 2009 © Ramesh Jain 56

57. ICIG Xian Sept 21, 2009 © Ramesh Jain 57

58. Sentient EventWeb
• Use sensors to create events.
– This conference
• Infrastructure to create and link all such events.
– Index
– Search
– Present
• Capture events and put them on the Web.
Imagine each sensor tweeting.
ICIG Xian Sept 21, 2009 © Ramesh Jain 58

59. Sentient EventWeb
• Use sensors to create events.
– This Seminar
• Infrastructure to create and link all such
events.
– Index
– Search
– Present
• Archive events and put them on the Web.
ICIG Xian Sept 21, 2009 © Ramesh Jain 59

60. First Version: E2E
Communication Modes:
• Person to Person
• Device to Device
– Telephone
– Video Conferencing
• Environment to Environment (E2E)
ICIG Xian Sept 21, 2009 © Ramesh Jain 60

61. Environment
• Fixed objects
• Active Objects
• Events and Activities
• Sensor to capture Objects and Events
– Cameras, microphones, pressure, RFID, …
• Displays and Actuators
– Projectors, Displays, Speakers, …
ICIG Xian Sept 21, 2009 © Ramesh Jain 61

62. Key Idea
• Put enough sensors and devices to make
mediation disappear.
• System detects events and objects and
sends appropriate data streams and
displays/renders incoming streams on
appropriate devices
• Bring back Person to Person
communication
ICIG Xian Sept 21, 2009 © Ramesh Jain 62

63. Vision
• First build Communication Nodes
• Next build Web Nodes
• Build Sentient EventWeb
ICIG Xian Sept 21, 2009 © Ramesh Jain 63

64. E2E communication: Vision
Environment 1 Environment 2
Device to Device
Sentient Sentient
Information
communication
Information
System Web
System
ICIG Xian Sept 21, 2009 © Ramesh Jain 64

65. Environment: Node Architecture
EventBas
Sensors e
Situation
Physical Environment Environment Network/
MMDB based
Environment Model Server Transmiss
controller
ion
Actuators /
Presentation Actuator /
Devices Presentation
Model
ICIG Xian Sept 21, 2009 © Ramesh Jain 65

66. Event detection and organization
ICIG Xian Sept 21, 2009 © Ramesh Jain 66

67. Joint Env. 2
Env.
1
SM
JSM
JSM
1
1
JSM 2 Env. 3
Env. 5
Env. 4
Shared Visualization Spaces for Environment to Environment Communication , in
Workshop on Media, Arts, Science Xian Sept 21, 2009 (MAST 09), 2009.
ICIG and Technology © Ramesh Jain 67

68. Telemedicine Application
ICIG Xian Sept 21, 2009 © Ramesh Jain 68

69. Conclusion
• New models for Events and Objects are
required for bridging the semantic gap.
• Semantics emerges due to
relationships captured in Web.
• EventWeb facilitates emerging
applications dealing with multimedia
data.
ICIG Xian Sept 21, 2009 © Ramesh Jain 69

70. Thanks for your time and attention.
For questions: jain@ics.uci.edu
ICIG Xian Sept 21, 2009 © Ramesh Jain 70