How to Get Context Data Out of Robots Webinar - 15th May 2019 Corresponding webinar recording: https://youtu.be/T8XV6Wah9l4 How to retrieve Context Data and actuate robots. Chapter: Robotics Difficulty: 2 Audience: Technical Domain Specific Presenter: Francisco Meléndez (Technical Expert & Evangelist, FIWARE Foundation)

1. FIWARE (Introduction to Robotics)
Francisco Melendez
Technical Expert and Evangelist
FIWARE Foundation e.V
francisco.melendez@fiware.org

2. Learning Goals
● Introduction to FIWARE :
○ What is Context Data? What is a Smart Solution?
○ How is Context Data used within Smart Solutions?
○ What is FIWARE? How does it help?
● How can FIWARE and Standardized Context Data Management create
synergies between Robotic Systems and other Smart Solutions?
● What is a Robotic System?
● How to get/put context data out from/into robotic systems?

3. What is Context Data?
● The next Digital revolution will transform the daily
life of people and businesses
● Everything that can be connected will be connected
for the sake of a more smart life

4. What is Context Data?
● The next Digital revolution will transform the daily
life of people and businesses
● Everything that can be connected will be connected
for the sake of a more smart life
● The emerging Digital Life gravitates around Context
Data, which describes what is going on, where,
when and why
● Every physical or virtual context entity should be
able to answer the following questions:
○ Who are you? → Entity ID
○ What are you? → Entity type (Data Model)
○ What is your current state?→Entity attributes

5. What is a Smart Solution?
How is Context Data used in a Smart Solution?
Smart Solutions gather data from many different sources (including but not limited to IoT)
to build a “picture” of the real world and then process and analyze that information in order
to implement the desired intelligent behavior (which may imply changing the real world)

6. What is FIWARE?
FIWARE IS an open source initiative
defining a universal set of standards
for context data management
● It is free open-source technology
● It is a focused solution based
around context data
● It is standards driven
● It IS NOT a competitor of existing
IoT protocols, transport methods,
or domain-specific solutions

7. The FIWARE Catalogue
https://github.com/FIWARE/catalogue

8. The FIWARE community
www.fiware.org/community
7
An independent Open Community that aims to build an open sustainable ecosystem
around public, royalty-free and implementation-driven software platform standards that will
ease the development of new Smart Applications in multiple sectors”.
● Smart Cities: www.fiware.org/community/smart-cities
● Smart Industry: www.fiware.org/community/smart-industry
● Smart Agrifood: www.fiware.org/community/smart-agrifood
● Smart Energy: www.fiware.org/community/smart-energy
Success Stories → www.fiware.org/success_stories

9. What is Context Data? What is a Smart Solution?
Example 1

10. What is Context Data? What is a Smart Solution?
Example 1

11. ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
What is Context Data? What is a Smart Solution?
Example 1

12. ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
What is Context Data? What is a Smart Solution?
Example 1

13. ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
What is Context Data? What is a Smart Solution?
Example 1

14. ID: PM_xx
type: Parking Machine
Attributes:
Private Owner Data
(secured):
· Parking Place 1:
Car ID,
time paid for
· Parking Place 2:
Car ID,
time paid for
…
Public Data:
· Location
· Free/busy slots
· Current cost per
minute/hour
· Mean time to
find a free
parking slot
...
ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
What is Context Data? What is a Smart Solution?
Example 1

15. ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: PM_xx
type: Parking Machine
Attributes:
Private Owner Data
(secured):
· Parking Place 1:
Car ID,
time paid for
· Parking Place 2:
Car ID,
time paid for
…
Public Data:
· Location
· Free/busy slots
· Current cost per
minute/hour
· Mean time to
find a free
parking slot
...
What is Context Data? What is a Smart Solution?
Example 1

16. What is Context Data? What is a Smart Solution?
Example 1
ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
ID: PM_xx
type: Parking Machine
Attributes:
Private Owner Data
(secured):
· Parking Place 1:
Car ID,
time paid for
· Parking Place 2:
Car ID,
time paid for
…
Public Data:
· Location
· Free/busy slots
· Current cost per
minute/hour
· Mean time to
find a free
parking slot
...

17. What is Context Data? What is a Smart Solution?
Example 2

18. What is Context Data? What is a Smart Solution?
Example 2

19. RED
What is Context Data? What is a Smart Solution?
Example 2

20. Perception
Components
What is Context Data? What is a Smart Solution?
Example 2

21. Perception
Components
What is Context Data? What is a Smart Solution?
Example 2

22. Perception
Components
What is Context Data? What is a Smart Solution?
Example 2

23. Perception
Components
What is Context Data? What is a Smart Solution?
Example 2

24. BEHAVIOUR
Data Fusion,
Deliberative,
and
Executive
Components
Perception
Components
What is Context Data? What is a Smart Solution?
Example 2

25. Developing a robust, standard perception
mechanism for “red traffic light states” is
extremely challenging from the robotic
perspective…
● light alters color perception
● heterogeneous traffic light
locations, structures and shapes
● Occlusions
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
BEHAVIOR
What is Context Data?
Example 2

26. Developing a robust, standard perception
mechanism for “red traffic light states” is
extremely challenging from the robotic
perspective…
● light alters color perception
● heterogeneous traffic light
locations, structures and shapes
● Occlusions
● Mimicking the human does not ensure
the desired robot behavior
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
BEHAVIOR
What is Context Data? What is a Smart Solution?
Example 2

27. Developing a robust, standard perception
mechanism for “red traffic light states” is
extremely challenging from the robotic
perspective…
● light alters color perception
● heterogeneous traffic light
locations, structures and shapes
● Occlusions
● Mimicking the human does not ensure the
desired robot behavior
In this problem, sharing context data between
geo-located mobile robots and smart traffic lights
reduces the problem uncertainty, the situation
“RED traffic light” is now a fact and allows
the robot’s decision mechanisms to command
the desired robot behavior
Context
Aware
Components
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
BEHAVIOR
CONTEXT
DATA
SMART CITY
traffic light
system
RED
What is Context Data? What is a Smart Solution?
Example 2

28. CONTEXT
DATA
Context
Aware
Components
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
SMART CITY
traffic alert
rule-based
system
(TARS)
SMART CITY
traffic light
(TL)
SERVICE
Semantically
Annotated
City Maps
What is Context Data? What is a Smart Solution?
Example 2
DESIRED
BEHAVIOR

29. Id: ROBOT_id
Type: Delivery Robot
Attributes:
(data out)
Location: (x,y)
Target: (x’, y’)
Subscriptions:
TARS Alerts
(data in)
Alert:
Location_Type:
pedestrian crossing
Traffic Light:
Present
State:
RED
Recommended Actions:
CONTEXT
DATA
(BROKER)
Context
Aware
Components
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
Id: TL_id
Type: Traffic Light
Attributes:
(data out)
Location: (x,y)
State: RED
Id: TARS_id
Type: Traffic alert
Rule-Based System
Attributes:
(data out)
Subscribed to:
TL_id
Alerts: (Robot_id,
attributes…)
(data in)
Subscription
Requests:
( Robot_id, x0 y0 , x’
y’ )
…
SMART CITY
traffic alert
rule-based
system
(TARS)
SMART CITY
traffic light
(TL)
SERVICE
Semantically
Annotated
City Maps

30. Id: ROBOT_id
Type: Delivery Robot
Attributes:
(data out)
Location: (x,y)
Target: (x’, y’)
Subscriptions:
TARS Alerts
(data in)
Alert:
Location_Type:
pedestrian crossing
Traffic Light:
Present
State:
RED
Recommended Actions:
CONTEXT
DATA
Context
Aware
Components
Perception
Components
Data Fusion,
Deliberative,
and
Executive
Components
Id: TL_id
Type: Traffic Light
Attributes:
(data out)
Location: (x,y)
State: RED
Id: TARS_id
Type: Traffic alert
Rule-Based System
Attributes:
(data out)
Subscribed to:
TL_id
Alerts: (Robot_id,
attributes…)
(data in)
Subscription
Requests:
( Robot_id, x0 y0 , x’
y’ )
…
SMART CITY
traffic alert
rule-based
system
(TARS)
SMART CITY
traffic light
(TL)
SERVICE
Semantically
Annotated
City Maps

31. Real World
ENTITIES
What is a Robot?
Sensors Actuators
ROBOTIC SYSTEM
READINGS
COMMANDS
Integrated Planning and
Control abilities
(Think)
Sense
Think
Act

32. Real World
ENTITIES
What is a Robot?
Sensors Actuators
ROBOTIC SYSTEM
READINGS
COMMANDS
Integrated Planning and
Control abilities
(Think) (Sense)
(Think)(Act)
Sense
Think
Act

33. OTHER
ENTITIES
Sensors Actuators
ROBOTIC SYSTEM
READINGS
COMMANDS
Integrated Planning and
Control abilities
(Think)
What is a Robot?
Robotic Systems as Context Entities (I) → Black Box View
Action Specification
Action
Parametization
Action Monitoring and
Control
Task Specification
Task Parametization
Task Monitoring and
Control
Localization
Mapping
Motion Constraints
Interfaces
CONTEXT DATA
· ID
· Robot Type
· Attributes:
Current State
and
Configuration
of
Layers/Interfaces
Sensor Readings
Actuator Commands
Actuator Commands

34. Real World
ENTITIES
Sensors Actuators
Controllers (Executive)
Deliberative Layer 1
Deliberative Layer 2
Deliberative Layer ...Deliberative Layer ...Deliberative Layers ...
Cognitive Layer
ROBOTIC SYSTEM
READINGS
COMMANDS
What is a Robot?
Robotic Systems as Context Entities (II) → Layered Model View
Robotic systems are complex

35. Robotic systems are complex
Layers often encapsulate a sub-structure of
sense-think-act models
Each layer produces and consumes
context data at a different level of abstraction
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Real World
ENTITIES
Sensors Actuators
Controllers (Executive)
Deliberative Layer 1
Deliberative Layer 2
Deliberative Layers...
Cognitive Layer
ROBOTIC SYSTEM
READINGS
COMMANDS
What is a Robot?
Robotic Systems as Context Entities (II) → Layered Model View

36. Every layer integrates perception, planning, and control
abilities to fulfill its purpose but, in general:
● Top layers tend to emphasize on planning and semantics
● Bottom layers tend to emphasize on control and sensing
The lowest layer (executive) integrates the critical control loop.
At this level, the robust and efficient performance of a purpose-
specific robotic software platform is key to meet the safety and
real-time requirements of robotic applications.
Standardized context data management is effective to:
● monitor robot software platform states
● feed smart applications with data generated by the robot’s
perception mechanisms
● deploy, update, and configure robotic software
● increase the context awareness of middle and top layers
to support the robot’s decision mechanisms
Real World
ENTITIES
Sensors Actuators
Controllers (Executive)
Deliberative Layer 1
Deliberative Layer 2
Deliberative Layer ...Deliberative Layer ...Deliberative Layers ...
Cognitive Layer
ROBOTIC SYSTEM
READINGS
COMMANDS
What is a Robot?
Robotic Systems as Context Entities (II) → Layered Model View

37. Real World
ENTITIES
Actuators
Executive Layer
(Action Level)
Deliberative Layer 1
(Task Level)
ROBOTIC SYSTEM
Deliberative Layer 2
(Mission Level) Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sensors
READINGS
COMMANDS
Planning
Control
What is a Robot?
Robotic Systems as Context Entities (II) → Layered Model View
Smart
Decisions
Safety,
Robustness
and
Efficiency
Long-term
Goals
Accuracy
of
sensor
readings
and
actuator
Commands

38. Example: Layered Model
(Missions, Tasks, Actions)
Real World
ENTITIES
Sensors Actuators
Executive Layer
(Action Level)
Deliberative Layer 1
(Task Level)
ROBOTIC SYSTEM
(Layered Model)
Deliberative Layer 2
(Mission Level)
MISSION LAYER
(e.g. Product Delivery)
Task Coordination
and
Level 2 Interface
TASK LAYER
(e.g. Navigation A → B)
Task Planning,
Execution Monitoring
and
Level 1 Interface
ACTION
(e.g. Path Planning,
Obstacle Avoidance)
Action, Perception
and
Level 0 Interface
Action
Specification
Action
Parametization
Action Monitoring
and Control
Task Specification
Task
Parametization
Task Monitoring
and Control
Localization
Mapping
Motion
Constraints
Algorithms’,
Configuration Space’,
System states’,
Actuator Commands
and Sensor Readings
Algorithms’’,
Configuration Space’’,
System States’’,
Action Primitives,
Action Pre-Conditions,
and Action Effects
Algorithms’’’,
Configuration Space’’’,
System States’’’,
Task Primitives,
Task Pre-Conditions,
and Task Effects
Layer CompositionLayer Name / Role Interfaces
CONTEXT DATA
· ID
· Robot Type
· Attributes:
Current State
and
Configuration
of
Layers/Interfaces

39. Real World
ENTITIES
Sensors Actuators
Controllers (Executive)
Behavior
1
READINGS
COMMANDS
Behavior
2
...
Behavior Coordination
Deliberative Layer 1
(Task Level)
Deliberative Layer 2
(Mission Level)
Executive Layer
(Action Level)
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
C1 C2
Sense
Think
Act
Sense
Think
Act
Sense
Think
Act
C3
C4
C5
What is a Robot?
Robotic Systems as Context Entities (III) → BEHAVIOR/COMPONENT views
Real World
ENTITIES
Sensors Actuators
READINGS
COMMANDS

40. BLACKBOX LAYER-BASED BEHAVIOR-BASED COMPONENTS
● Reference Robotic System Patterns:
Sense
Think
Act
● Goal of your Application’s
Context-Data Loop:
STANDARDIZED CONTEXT DATA MODELS AIM TO SUPPORT:
· The deployment and maintenance of Robotic Software
· The monitoring,
processing,
and update of
Robot States,
Configurations
and Interfaces
➔Robotic System as Black Boxes
➔Robotic Layers, Behaviors or Components
➔Human-Machine and Machine-Machine Interfaces
➔Application-Specific features for Data Fusion, Planning,
Perception, Execution, Navigation and Manipulation
● Data Model Design

41. Robotic System
ORION
CONTEXT BROKER

42. What is a Robot?
Robotic Systems as Context Entities (IV) → Reference Implementations
Legacy/Proprietary
Robotic System
Legacy/Proprietary
Robotic System
ROS ROS 2
ROS
Robotic System
ROS 2
Robotic System
Resource
Constrained
Robotic System
(micro-controllers)
ORION
CONTEXT BROKER

43. ROS 2
Robotic System
What is a Robot?
Robotic Systems as Context Entities (IV) → Reference Implementations
Legacy/Proprietary
Robotic System
Legacy/Proprietary
Robotic System
ROS ROS 2
ROS
Robotic System
ROS 2
Robotic System
Resource
Constrained
Robotic System
(micro-controllers)
ORION
CONTEXT BROKER
OPC-UA Agent FIROS FAST-RTPS FIROS FAST-RTPS
micro-ROS
Agent
FAST-RTPS

44. Links
● Legacy or Proprietary Software
○ https://www.fiware.org/fiware-webinars/
■ Webinar 1: What is an IoT Agent?
■ Webinar 2: How to Debug IoT Agents
○ OPC UA Agent: https://iotagent-opcua.readthedocs.io/en/latest/
● ROS 1
○ http://www.ros.org/
○ FIROS https://github.com/iml130/firos
● ROS 2
○ https://index.ros.org/doc/ros2/
○ Fast RTPS https://eprosima-fast-rtps.readthedocs.io/en/latest/
● ROS2 for micro-controllers
○ https://micro-ros.github.io/

45. ID: FDR_xxx,
Type: Food Delivery Robot
Attributes:
- Private Delivery Company Data
(secured)
· Customer Details
· Current order
details
· Time to delivery
- Private Robot Status
(secured)
· Semantic City Maps
· Current Location
· Target Location
· Navigation Route
· Content
· Speed
· Battery Level
- Public Data
· Company Name
· Address
· Link to Ordering Web Page/App
· List of Services/Products
· Offers
ID: PM_xx
type: Parking Machine
Attributes:
Private Owner Data
(secured):
· Parking Place 1:
Car ID,
time paid for
· Parking Place 2:
Car ID,
time paid for
…
Public Data:
· Location
· Free/busy slots
· Current cost per
minute/hour
· Mean time to
find a free
parking slot
...
ID: SP_2
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession
How to get context data out from / put context data into Robots?
Application-Oriented View
ID: SP_1
type: Smartphone
Attributes:
Private Owner Data
(secured):
· Owner’s Identity
· Current Location
· Day Schedule
...
Public Data:
· Interests
· Social Media
· Profession

46. COMPONENTSBEHAVIORSLAYERSBLACKBOX
● Reference Robotic System Implementations
Sense
Think
Act
● Goals of your Context-Data Loop
STANDARDIZED CONTEXT DATA MODELS AIM TO SUPPORT:
· The deployment and maintenance of Robotic Software
· The monitoring,
processing,
and update of
Robot States,
Configurations
and Interfaces
➔Robotic System as Black Boxes
➔Robotic Layers, Behaviors or Components
➔Human-Machine and Machine-Machine Interfaces
➔Application-Specific features for Data Fusion, Planning,
Perception, Execution, Navigation and Manipulation
● Data Model Design
How to get context data out from / put context data into Robots?
Robot-Oriented View
● Reference Robotic System Patterns

47. Thank you!
http://fiware.org
Follow @FIWARE on Twitter