This tutorial was given in DEBS 2015, on June 29, 2015. The conference was held in Oslo.

1. When Artificial Intelligence meets the Internet of
Things. DEBS’15 tutorial
Speaker: Opher Etzion

2. 2
The autonomous car
Needs sensors for observing what happens now,
needs intelligence to understand what it
observes, needs intelligence to drive, needs
actuators to carry out the driving….

3. 3
Like the human body, we need to sense, to make
sense of what we sense, to make constant
decisions and to carry them out.
Sensing
Making
sense
from the
sensing
Real-time
decision
making
Acting

4. OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
4
Topic I
TOPIC II
Topic III
Topic IV

5. OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
5
Topic I
TOPIC II
Topic III
Topic IV

6. None of the
authorized drivers
location is near the
car’s location
theft is concluded
Use a built-in car
stopper to slow the
intruder and dispatch
the security company
A person enters a
car and the car
starts moving;
the person does not
look like one of the
authorized drivers
Such applications
become possible
since everything is
connected
6

7. 7
The term “Internet of Things” was coined by Kevin
Ashton in 1999.
His observation was that all the data on the
Internet has been created by a human.
His vision was: “we need to empower computers with
their own means of gathering information, so they
can see, hear, and smell the world by themselves”.

8. 8
The world of sensors
1 Acoustic, sound, vibration
2 Automotive, transportation
3 Chemical
4 Electric current, electric potential,
magnetic, radio
5 Environment, weather, moisture,
humidity
6 Flow, fluid velocity
7 Ionizing radiation, subatomic particles
8 Navigation instruments
9 Position, angle, displacement, distance,
speed, acceleration
10 Optical, light, imaging, photon
11 Pressure
12 Force, density, level
13 Thermal, heat, temperature
14 Proximity, presence

9. 9
The value of sensors
Kevin Ashton: “track and count everything, and
greatly reduce waste, loss, and cost. We could
know when things needs replacing, repairing or
recalling, and whether they were fresh or past
their best”
The value is in the ability to know and react in a
timely manner to situations that are detected by
sensors

10. 10
Differences between the traditional Internet to
the Internet of Things
Topic Traditional Internet Internet of Things
Who creates content? Human Machine
How is the content
consumed?
By request By pushing information
and triggering actions
How content is
combined?
Using explicitly defined
links
Through explicitly
defined operators
What is the value? Answer questions Action and timely
knowledge
What was done so far? Both content creation
(HTML…) and content
consumption (search
engines)
Mainly content creation

11. Two separate but connected goals:
Awareness and Reaction
Awareness Reaction
Event
Detect Derive Decide Do

12. Detect
Some
Noun of
Importance but
different
The act of bringing into a system’s sphere of understanding knowledge
about an event.
The detection is done by sensors, instrumentation and human
reports.
SwimLane
TriggerEvent
Activity
State
Change

13. 13
Intelligent Detection
Determine what is actually been sensed:
vision understanding, voice understanding,
text understanding.

14. Derive
The act of becoming aware of events that are not directly detectable
by bringing together events with other events, data, patterns and
publishing the observation as a derived event.
Raw
events
Raw
events
Raw
events
A Person or a computer recognizes the pattern and
enters the
derived event or just reacts to it directly.

15. 15
Event processing: making sense from what we
sense…
Combining data from multi-sensors to get
observations, alerts, and actions in real-time gets
us to the issue of detecting patterns in event
streams

16. 16
Intelligent derivation
Find the causality between events and
situations. We discuss the notion of
causality later.

17. Decide
The act of determining the course of action to do in response to the
situation. This includes the background information needed to be
collected to make the decision.
Pass through: Sometimes there is no decision. There is only
one course of action.

18. 18
Intelligent Decision
Finding the best decision some times under
real-time constraints may require an
intelligent process.

19. Do
The act of performing the course of action that was decided upon.
Notification: Sending a signal of sort to either a person or
system. This would include calling a web-service or
subscription to alerts.

20. 20
Intelligent Actions
Intelligent actuators

21. 21
Knowledge acquisition for IoT based systems
How do we know how to
make sense of all
these data?

22. OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
22
Topic I
TOPIC II
Topic III
Topic IV

23. 23
IoT and robotics
Robots serve as intelligent actuators

24. 24
Healthcare robotics
Rehabilitation robots: enhancing patients
with motoric and cognitive skills
Assistive robots: Robots for independent
living of disabled persons

25. 25
Some future healthcare
robotics applications
Automated assistance of monitored
patients

26. 26
Some future healthcare
robotics applications
Help in sit-to-stand and sit-down actions
for people with motor disabilities

27. 27
Some future healthcare
robotics applications
Autonomous moving of drugs and medical
equipment within the hospital

28. 28
Some future healthcare
robotics applications
Support of medical staff in various
activities

29. 29
Some future healthcare
robotics applications
People movement and movement monitoring

30. 30
Some future healthcare
robotics applications
People assistance in panic and danger
situations

31. 31
I(
The classical use of robots are for
industrial purposes: production,
machinery control, product design…
Industrial Robots

32. 32
I(
Autonomic management and coordination of
production activities among multiple
robots
Industrial Robots and
IoT

33. 33
I(
Autonomous management of equipment and
instruments
Industrial Robots and
IoT

34. 34
I(
Immediate reaction to critical situations
such as: high temperature, harmful
chemicals in the air
Industrial Robots and
IoT

35. 35
I(
Autonomic control of electrical and energy
plants
Industrial Robots and
IoT

36. 36
Robotics for defense
Robots are used for unmanned tools
(ground and air) for transport and
intelligence , threat detection and combat

37. 37
Robotics for defense and IoT
Autonomous and smart detection of
harmful chemicals and biological weapons

38. 38
Robotics for defense and IoT
Autonomic control of land vehicles and
aircrafts

39. 39
Robotics for defense and IoT
Identification and access prevention of
suspicious people intruding to sensitive
places

40. 40
Robotics for defense and IoT
Rescue trapped people

41. 41
The Internet of things for the elderly
and healthcare in general

42. 42
Safety sensors
Motion
sensor
Door
sensor
Chair
Sensor
Voice
Sensor
Alert
family
member
Alerts example:
Door was not locked within 2 minutes after
entrance
Falling event detected
Vocal distress detected
No motion for certain time period detected

43. 43
Medical sensors for the elderly

44. 44
E-Health sensors
Personalized alerts based on collection of monitors

45. 45
Pre-mature babies monitoring
Personalized alerts based on collection of
monitors: when nurse should be alerted, when
physician should be alerted.
There are many false alerts that are ignored,
Missing or ignored alert is sometimes fatal

46. 46
Track everything in a hospital

47. 47
Track the progress of a surgery relative to the plan
Detect significant deviation from plan that requires
rescheduling and trigger real-time rescheduling of
surgeries, assignments, and equipments.

48. 48
Dynamic planning
Example: traffic control; patient
treatment; serviceman scheduling

49. 49
AI meets IoT – Apple’s Perspective
Siri was released as Apple’s “intelligent
personal assistant”.
A sensor enabled Siri is targeted as a
“smart home solution”

50. 50
AI meets IoT – Google’s Perspective
Google acquired a collection of IoT related
companies and then acquired AI company
DEEPMIND that uses Neural Nets and
Reinforcement learning. The aim is to
develop a machine with intelligence of a
toddler with IoT providing sensing
capabilities

51. 51
AI meets IoT – Facebook’s Perspective
Facebook acquires wit.ai – a speech
recognition company. Making the
Internet of Things voice controlled.

52. OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
52
Topic I
TOPIC II
Topic III
Topic IV

53. 53
Vision understanding
Robocup tournament: Robots playing
football. Strong vision capabilities are
required.

54. 54
Vision understanding
Tracking objects over time from a
collection of cameras

55. 55
Vision understanding
Grace, the robot, can communicate with her
surrounding, understand gestures,
attended conferences, understands that
she had to stand in a line, go in an
elevator and ask people to press the floor
number…

56. 56
Speech recognition
Acoustic analysis, linguistic Interpretation

57. 57
Causality
In order to derive situations from events
there is a need to identify causalities.
Statistical methods can infer correlations.
Causality inference is more tricky….

58. 58
Causalities in events
Type I: predetermined causality - Event E2
always (or conditionally) occurs as a
result the occurrence of E1, thus we
don't need to have any sensor to detect
event E2 we may assume it happened if E1
happened (and the condition is satisfied),
some time offset or interval may be
attached to this causality. Note that in
this case E1 and E2 are both raw events.
Necessity and relevance

59. 59
Causalities in events
Type II: The event E1 is an input to a
processing element PE and event E2 is an
output of PE. In this case E2 is a derived
(virtual) event.
The specification of PE is part of the
system, thus the context and conditions
are known.
Necessity and relevance

60. 60
Causalities in events
Type III: The event E1 is an event that is
sent from a computerized system to a
consumer C. C applies (conditionally) some
action AC, where the specification of AC
is not known to us, but we observe that it
emits the event E2. This is another type
of causality (the event E2 would not have
been emitted, if E2 would not have
triggered AC), however, E2 may or may
not have functional dependency with
respect to E1
Necessity? and relevance?

61. 61
Causal inference
How the knowledge about causality is being
acquired?
Expert knowledge
Statistical inference
Inference using semantic or association net
Necessity? and relevance?

62. 62
Dangers of using correlation as causality
indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three
interpretations
The faster windmills are observed to
rotate, the more wind is observed to be.
Therefore wind is caused by the rotation of
windmills.

63. 63
Dangers of using correlation as causality
indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three
interpretations
Sleeping with one's shoes on is strongly
correlated with waking up with a
headache.
Therefore, sleeping with one's shoes on
causes headache.
(correct answer: going to bad drunk causes
both)

64. 64
Dangers of using correlation as causality
indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three
interpretations
As ice cream sales increase, the rate of
drowning deaths increases sharply.
Therefore, ice cream consumption causes
drowning. (real answer: they are both in
the same context – summer).

65. Temporal indeterminacy
Inexact indicator Probability
Event did not occur 0.4
Event occurred before T1 0.1
Event occurred in [T1, T2] 0.45
Event occurred after T2 0.05
T1 T2

66. False positives and negatives
False positive:
The pattern is matched;
The real-world situation
does not occur
False negative:
The pattern is not matched;
The real-world situation
occurs
Learning
from
experience

67. 67
Data is not good enough…

68. 68
Real-time decision under uncertainty
Robust RT
Optimization
Stochastic RT
Optimization
Simulation-based
RT optimization

69. Handling event uncertainties
Uncertain whether an
reported event has occurred
(e.g. accident)
Uncertain what really
happened. What is the type
and magnitude of the
accident (vehicles involved,
casualties)
Uncertain when an event
occurred (will occur): timing
of forecasted congestion
Uncertain where an event
occurred (will occur):
location of forecasted
congestion
Uncertain about the level of
causality between a car
heading towards highway
and a car getting into the
highway
Uncertain about the accuracy
of a sensor input: count of
cars, velocity of cars…
The pattern: more
than 100 cars
approach an area
within 5 minutes after
an accident derives a
congestion forecasting
Uncertain about the validity
of a forecasting pattern
Uncertain about the quality
of the decision about traffic
lights setting

70. Predictive Event Processing (1)
VS.
Photo by Michael Gray, Flickr

71. Predictive Event Processing (2)
VS.
+

72. Predictive Event Patterns
Pattern  Future event, probability, time interval
“4 high value deposits from different geographic locations within 3 days”
 “0.6 chance for a large transfer abroad, in 1 day”
“Output event will occur with
distribution D over interval (t1,t2)”
Stock decrease of > 5% in 3 hours
 Good chance for 2% increase within 2 hours

73. Limitations of the use of rules in specifying predictive event
patterns
Limitations:
1. Partial patterns
2. Uncertain input events
3. Complex relationship between random variables
Rule = hard-coded probabilistic
Relationship

74. Dynamic event prediction
Time Series
Prediction
Graphical models
Temporal Graphical models

75. Learning patterns and causalities
Event
Patterns
Pattern and causality acquisition
This is a direction
to reduce the
complexity of
application
development
There are challenges in doing it – since “detected situations”
are “inferred events” and may not be reflected in past events

76. 76
Security challenges of IoT
Getting security feeling is a necessary condition for the success of
IoT to become pervasive.

77. 77
Dangers and challenges
Confusing a sensor
Changing the rules of the game
Abusing an actuator

78. 78
Security considerations of IoT
Murder by the Internet
“With so many devices being Internet connected, it makes murdering
people remotely relatively simple, at least from a technical
perspective. That’s horrifying,” said IID president and CTO Rod
Rasmussen. “Killings can be carried out with a significantly lower
chance of getting caught, much less convicted, and if human history
shows us anything, if you can find a new way to kill, it will be
eventually be used.”
EXAMPLES: Turn off pacemakers, Shutdown car systems while
driving, stop IV drip from functioning

79. 79
Confusing a sensor
The same as confusing the human
eyes.
See things that don’t exist, don’t
see things that exist, distort
picture…

80. 80
Confusing a sensor
Can be used to sabotage anti-
crime systems, to commit fraud,
or just damage something or
someone…

81. 81
Confusing a sensor
Example from another domain: the
Twitter hoax

82. 82
Changing the rules of the game
The logic is
rule-based.
The ease of
modification
can be
abused to
add/delete/
modify rules,
change
thresholds…

83. 83
Changing the rules of the game
Changing data relevant for the
system: maps, pictures, person’s
data…

84. 84
Abusing actuators
Deviating
from
course,
shutting
down,
activating in
wrong
mode…

85. OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
85
Topic I
TOPIC II
Topic III
Topic IV

86. TOPIC 4
A futuristic view of the Internet of Things
following Ray Kurzweil’s predictions:
86

87. 87
Driverless car
Sensors that replace the human driver’s sensing,
and actuators that drive the car.
2017

88. 88
Automated
personal
assistant
Sensors that determine the context serves as
active advisors. They understand your context and
even listen to your conversations and give you
suggestions of what to say (e.g. through google
glass).
2018

89. 89
Computing
implants inside
the human
body
Sensors and actuators that fight any disease,
operate in the level of cell, and reprogram the
body to stop the aging process.
2020
2040
Short term: switch off our fat cells
Longer term: stay
young forever

90. 90
May 14, 2014

91. 91
Summary: The Internet of Everything participates in many of
the predictions about the future, including Kurzweil’s
singularity.
The responsibility is upon us to create this future…

92. 92
My main motivation is to use the experience and
knowledge I have accumulated over the years to make a
better world