Multimedia and Big Data are closely related topic. Big data enables solving some important challenges in multimedia and basic principles of multimedia are the key issues in multimedia.

1. 1
BAMMF
June
20,
2014

2. • Marshal
McLuhan
in
1964.
– A
medium
itself,
not
the
content
it
carries,
should
be
the
focus
of
study.
– A
medium
affects
the
society
in
which
it
plays
a
role
not
only
by
the
content
delivered
over
the
medium,
but
also
by
the
characteris@cs
of
the
medium
itself.
BAMMF
June
20,
2014
2
Media

3. Mul+
BAMMF
June
20,
2014
3
Media
+
+
Digital
technology
allowed
combining
and
using
media
leading
to
a
major
disrupAon.

4. Photos
are
the
killer
content
type
on
mobile.
Quick
to
consume
like
text,
but
easier
to
produce
on
a
phone.
BAMMF
June
20,
2014
4
From
Mary
Meeker’s
presenta@on

5. Big
Data
BAMMF
June
20,
2014
5

6. To
know
our
world,
(MulAmedia)
Data
is
the
Source.

7. McLuhan
was
right.
UnAl
20th
Century
-­‐-­‐
Medium
was
the
message.
BAMMF
June
20,
2014
7

8. BAMMF
June
20,
2014
8
What
is
an
Elephant?

9. BAMMF
June
20,
2014
9
Six
Blind
Men
Exploring
an
Elephant.

10. BAMMF
June
20,
2014
10
An
Elephant
according
to
Six
Limited
PerspecAves.
The
Blind
Men’s
Elephant

11. BAMMF
June
20,
2014
11
But,
an
Elephant
is
an
Elephant.
A
Real
Animal.

12. MulAmedia
and
Big
Data
are
both
the
Latest
Elephant.
BAMMF
June
20,
2014
12
Lots
of
people
excited
about
these
but
with
narrow
perspecAves.

13. BAMMF
June
20,
2014
13

14. Two
Major
Human
AcAviAes
• CommunicaAons
• ComputaAons
BAMMF
June
20,
2014
14
Both
require
MulAmedia
AND
Big
Data.

15. CommunicaAon
BAMMF
June
20,
2014
15

16. Nature
Gave
Us
Five
Basic
Senses
BAMMF
June
20,
2014
16
to
Experience
the
World

17. We
promptly
created
Silos
BAMMF
June
20,
2014
17

18. Storytelling:
The
Oldest
and
most
powerful
Art.
• A
story
is
a
chain
of
events
that
begins
at
one
place
and
ends
at
another.
• A
story
is
a
selecAve
batch
of
informaAon.
It
selects
details,
arranges
them,
perhaps
embellishes
them.
BAMMF
June
20,
2014
18
h6p://janefriedman.com/2011/09/27/what-­‐is-­‐a-­‐story/

19. Where
do
Events
happen?
• In
Real
World.
• Images,
video,
audio,
all
other
sensors
are
just
a
means
to
detect
and
capture
some
aspect
of
the
event.
BAMMF
June
20,
2014
19

20. Events
are
‘Connectors’
Events
create
‘Context’
Event
contains
Information
and
Experiences
Spatial
Causal
Experiential
Informational
Temporal
Structural

21. So
What
is
Digital
Storytelling?
• Prepare
a
database
of
all
events.
– Collect
all
experienAal
and
informaAonal
data
about
each
event.
• Considering
audience
and
the
message,
select
important
events.
• Render
Events
– For
each
event
decide
what
informaAon
and
experienAal
data
should
be
used.
– Render
all
events
and
related
data
in
the
best
format
for
highest
quality
of
experience.
BAMMF
June
20,
2014
21

22. From
Events
to
Stories
CollecAng
Events
SelecAng
Events
Rendering
Events

23. Oral
Storytelling
BAMMF
June
20,
2014
23

24. Novels
BAMMF
June
20,
2014
24

25. Video
BAMMF
June
20,
2014
25

26. Do
you
edit
your
videos?
• Why
is
video
ediAng
so
difficult?
– Even
a_er
so
many
products?
• How
can
we
make
it
usable
by
consumers?
• Is
Video
ediAng
a
real
problem?
• Is
it
possible
to
reinvent
video
ediAng
given
all
the
‘big
data’?
BAMMF
June
20,
2014
26

27. Virage Video Indexing
Lyn Russell
00:08:26:12 00:08:34:29 00:08:40:00
00:08:26:12 00:08:34:29 00:08:40:00
“Clinton spoke
with reporters”
“When She arrived”
“Older citizens
who have”
Clinton running
for senate
Speaking
at rally
Improving
Medicare
Lyn Russell
Hillary ClintonFace ID
Speaker ID
User Annotation
Speech to Text
Time Code
Keyframes
Video
Time
Hillary Clinton

28. Different
Data
Tracks
Ac+vity
Loca+on
Manual
Visual
Audio
OTHER
Walk
Walk
Walk
Walk
Walk
Walk
Exercise
Drive
Home
Drive
Mee+ng
Work
Exercise
Home
Drive
Mee+ng
Work
Home
Mee+ng
Work
Exercise
Event
Track

29. Personalized
Storytelling:
Events
in
a
Football
Game.
BAMMF
June
20,
2014
29

30. All
communicaAon
is
story
telling.
Even
scienAfic
papers.
BAMMF
June
20,
2014
30

31. Fundamental
Opportunity
in
MulAmedia
CommunicaAon
and
CompuAng:
UAlize
complementary
and
correlated
informaAon
from
mulAple
sources
to
form
holisAc
picture
of
the
world.
BAMMF
June
20,
2014
31

32. ComputaAon:
at
least
50,000
years
• CounAng
was
primarily
used
to
keep
track
of
social
and
economic
data
such
as
number
of
group
members,
prey
animals,
property,
or
debts.
• CompuAng
was
invented
for
Resource
management.
BAMMF
June
20,
2014
32

33. What
is
Important
in
‘Big
Data’?
MulAmedia
RealAme
Uncertainty
BAMMF
June
20,
2014
33

34. Big
data
is
the
Rorschach
blot
of
our
Ames
–
an
incomprehensible
shape
on
to
which
we
project
our
dreams
and
nightmares,
hopes
and
fears.
BAMMF
June
20,
2014
34
Rorschach
blot

35. BAMMF
June
20,
2014
35
You
can
only
control
what
you
measure.

36. The most Fundamental Problem
in Society:
Connecting People to Resources
Effectively, Efficiently, and Promptly
in given Situations.
36
BAMMF
June
20,
2014

37. Current
Social
Networks
Important
UnsaAsfied
Needs
37
Time
to
Focus
BAMMF
June
20,
2014

38. What
differenAates:
• Advanced
SocieAes
from
Primi6ve
SocieAes
• Advanced
countries
from
a
Developing
countries
38
BAMMF
June
20,
2014
How
People’s
needs
are
connected
to
Resources.

39. Hunter
Gatherer
Food
delivery
where
you
are.
39
BAMMF
June
20,
2014

40. UnAl
2000,
you
went
to
a
phone.
Now
the
call
comes
to
you.
40
BAMMF
June
20,
2014

41. UnAl
recently,
you
were
a
folder.
Very
Soon
You
are
Your
Data.
41
BAMMF
June
20,
2014

42. Needs
Need = f (persona, situation)
BAMMF
June
20,
2014
42

43. Where do most Data Scientists apply their Magic?
We
live
in
a
targeted
world.
AdverAsers
need
efficient
ways
to
si_
through
the
volume
of
data.
BAMMF
June
20,
2014
43

44. Using BIG Data to solve Big PROBLEMS
Time
is
right.
BAMMF
June
20,
2014
44

45. We
Live
in
Dynamic
World.
Big
Data
will
keep
gegng
bigger
in
Cyber
Space.
BAMMF
June
20,
2014
45

46.
Animals
Machines
SocieAes
Published
first
in
1942

47. • Desired
state
(Goal)
• System
model
and
Control
Signal
(AcAons)
• Current
State
(using
mulAmedia
data)

48. EventShop : E. Situation Detection
Predic+ve
Situa+on
Recogni+on
Evolving Global Situation
Predic+ve
Personal
Situa+on
Recogni+on
Personal EventShop
Evolving Personal Situation
Need- Resource Matcher
Recommendation
Engine
Persona
Database
Resources
Needs
Data
Inges+on
Wearable Sensors
Calendar
Location….
DataSources
….
Data
Inges+on
and
aggrega+on
Database Systems
Satellite
Environmental
Sensor Devices
Social Network
Internet of Things
Actionable Information
BAMMF
June
20,
2014
48

49. CriAcal
Requirements:
• Environmental
SituaAon
RecogniAon.
– Detects
geo-­‐spaAal
situaAons.
– Space
is
a
key
dimension.
– Helps
in
idenAfying
resources
and
connecAng
to
them.
• Personal
SituaAon
RecogniAon.
– Centered
around
an
individual.
– Uses
environmental
situaAon.
– Helps
in
modeling
individuals
and
idenAfying
their
needs.
BAMMF
June
20,
2014
49

50. Data
to
Situa+ons
Situa+ons
Event
Streams
Mul+modal
Observa+ons
Low-­‐level
Analysis
Aggrega+on
and
Classifica+on
BAMMF
June
20,
2014
50

51. A
Grand
Challenge
for
data
has
always
been:
Sense
making
from
mulAmodal
massive
geo-­‐social
data-­‐streams.
51
BAMMF
June
20,
2014

52. A
REAL
Challenge
• SituaAon
Modeling
• SituaAon
RecogniAon
(Object
recogniAon,
event
recogniAon,
speech
recogniAon,
…
dominated
last
century.)
BAMMF
June
20,
2014
52

53. Challenge:
Unifying
MulAmodal
Big
Data
• Spa+o-­‐temporal-­‐thema+c
(STT)
real-­‐@me
streams
• E-­‐mage
as
a
unifying
GRID
representa+on
53
(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)
d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)
BAMMF
June
20,
2014

54. 54
Billions
of
data
sources.
Environment
for
SelecAng,
and
Combining
appropriate
sources
to
detect
situaAons.
PredicAon
for
Pro-­‐acAve
acAons
InteracAons
with
different
types
of
Users
BAMMF
June
20,
2014

55. EventShop
BAMMF
June
20,
2014
55

56. 6/20/14
56
Flood level - Shelter
Flood Level
Shelter
Twitter
Classify (Flood level - Shelter)

57. 6/20/14
57

58. The Creative Destruction of Medicine
CompuAng
Power,
Data
Universe
Social
Networking
Internet
Mobile
connecAvity,
Bandwidth
InformaAon
Systems
Imaging
Wireless
Sensors
Genomics
New
Medicine
CreaAve
DestrucAon
BAMMF
June
20,
2014
58
Old
Medicine

59. Health = Genetics + Lifestyle
BAMMF
June
20,
2014
59

60. Asthma Triggers
Indoor
Allergens
Outdoor
Allergens
Smoke
Air
PolluAon
Chemical
Irritants
Asthma/Allergies
BAMMF
June
20,
2014
60

62. Everyone’s Respiratory Health is Different
Disaster
Situa+on
Assimila+on
and
Control
Environmental
Resources
and
Historic
Data
Governmental
Agencies
Internet
of
Things
Social
Sources
Experts
Users
All
Users
are
not
EQUAL.
BAMMF
June
20,
2014
62

63. AcAviAes
CommunicaAons
Media
Persona:
Turning
Disassociated
Data
into
Meaningful
InformaAon
AGGREGATED
PERSONAL
ANALYTICS
Sensors
63
Persona
BAMMF
June
20,
2014

64. Wearable 2013:
Data Data Everywhere, nor any Insight to Use.
h6p://www.medhelp.org/
BAMMF
June
20,
2014
64

65. HEALTH
PERSONA
Logical
Sensor
Physiological
Sensors
Fitness
Tracking
Sensors
Life
Event
Kine+c
Event
Physiological
Event
Food
Event
Personicle
(Health) Persona Framework
Calendar
Ac+vity-­‐level
Heart
rate
Home
Drive
Mee+ng
Work
Exercise
Walk
Walk
BAMMF
June
20,
2014
65

66. Time
Foursquare
Personal
Calendar
Moves
Drive
Walk
Mee+ng
in
DBH
room
3233
Walk
Drive
Mee+ng
with
trainer
in
gym
Ac+vity
Level
Personicle
Personicle
Home
Drive
Walk
Mee+ng
Work
Walk
Drive
Exercise
Work
Laleh
at
University
Club
Ea+ng
dinner
BAMMF
June
20,
2014
66

67. Insight: Exercise Triggers My Asthma
• Red: “Severe asthma attack”.
• Orange: High activity level.
• Intense Exercise triggers asthma!
Personicle
Ac+vity
Stream
t1
t2
t3
t4
t5
BAMMF
June
20,
2014
67

68. UCI
Allergy/Asthma
App
Triggers
Recommendati
on
Stop
Jogging!
Symptoms
SHORTNESS
OF
BREATH
CHEST
TIGHTNESS
OR
PAIN
COUGHING
WHEEZING
How Do You
Feel?
BAMMF
June
20,
2014
68

69. PredicAve
Techniques
for
PrevenAve
Advise
69
BAMMF
June
20,
2014

70. From Persona to Societal Health
Health
Persona
of
different
people
Environmental
factors
Disease
outbreak
Disease
Modeling
Geographical
loca+on
specifica+ons
BAMMF
June
20,
2014
70

71. Use
ObjecAve
Massive
Volume
of
Personal
Data
for
Building
Beker
Disease
Models.
Persona and Societal Health
Use
Disease
Model
and
Personal
Data
for
Beker
Quality
of
Life.
71
BAMMF
June
20,
2014

72. Live
EventShop
and
CollaboraAon
• Live
EventShop
Demo
– h6p://auge.ics.uci.edu/eventshop/
• Current
Collaborators
– Cyber-­‐Physical
Cloud
Compu+ng
Project
• NICT,
NIST
– SLN4MOP
Project
• Sri
Lanka
Farmers;
Prof.
Ginige
in
Sydney
leading
– Open
Source
EventShop.
• HCL
– Other
partners
BAMMF
June
20,
2014
72

73. MulAmedia
and
Big
Data
• MulAmedia
was
always
Big
Data.
• Many
Major
Challenges
in
Big
Data
are
the
same
as
in
MulAmedia.
• Time
for
MulAmedia
community
to
Accept
and
Act.
BAMMF
June
20,
2014
73

74. Thanks
for
your
Ame
and
akenAon.
For
ques+ons:
jain@ics.uci.edu