A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems

ALMA MATER STUDIORUM – UNIVERSIT`A DI BOLOGNA
Computer Science and Engineering Department (DISI) – Cesena Campus, Italy
A Model and Platform for Building Agent-Based
Pervasive Mixed Reality Systems
Angelo Croatti, Alessandro Ricci
{a.croatti | a.ricci}@unibo.it
16th
PAAMS International Conference
June 22nd
, 2018 – Toledo (Spain)
A. Croatti (UNIBO) Augmented Worlds June 22, 2018 1 / 32

Mixed Reality will be Reality. . .
The future of human consciousness will be a hybrid aﬀair.
We will live and work in a ubiquitous environment,
where physical reality and a pervasive digital layer mix
seamlessly according to the logic of software.
This is Mixed Reality, and it will soon simply be reality!
— John Rousseau [Rou16]

Mixed Reality
Nowadays, a mainstream technology!
To be exploited for:
designing smart environments, blending physical and virtual objects
introducing new opportunities in supporting individual and cooperative
human activities
Allows for:
reshaping the environments where people work and live
rethinking the way in which they interact and collaborate
. . .

Virtual Continuum and Technologies

Toward (Agent-based) Pervasive Mixed Reality Systems
Goal Investigate how MR technologies can be put in synergy with
Pervasive Computing and Internet-of-Things (IoT) – as main
ingredients of modern smart environments

The role of Agents in MR
Agents (and MAS) can play a key role:
modelling and implementing holograms with autonomous behaviour
supporting the interaction of holograms with other holograms and
humans
featuring the observation of the physical world
A“new era”of smart environments, where autonomous cognitive
agents can
dynamically create and control holograms and virtual objects
control physical things and physical environment
interact with humans in an homogeneous cyber-physical system

Related Works
MiRa – Mixed Reality Agents [HCO+11] and AuRAs – Augmented
Reality Agents [CSHO14]
the focus is about agents having a representation in an MR
environment to become perceivable by humans
IVEs – Intelligent Virtual Environments [LA00, RPLJ+16]
Agents for context aware applications
Virtual Agents
Mirror Worlds [RPTC15, RTP+15]

Augmented Worlds
The Augmented World (AW) Vision
Deﬁning a single and uniform conceptual model
capturing main aspects of current augmentation technologies
allowing for the design of smart pervasive mixed reality software
systems
with an deep inspiration to agents programming paradigm
Allowing the design and the development of innovative smart
environments with an high level of abstraction
proactive autonomous/cognitive software agents interacting with real
things, physical space and its inhabitants

Augmented Worlds
Augmented World
A software application that
enriches the functionalities
of a particular physical
environment exploiting
augmented entities
The idea of AWs is inspired
by Mirror Worlds [RPTC15]
and A&A meta-model
[ORV08]

Augmented Worlds Conceptual Model
Conceptual Model Formalization
Agent Augmented World Augmented Entity
AW User App Hologram
Region
+name
+extension
+ name
+ position
+ orientation
+ extension
+ obs state
+ actions()
+ extension
Human User
run
perceive, interact with
render
enter
join, quit
define, track, untrack
create, do act, track, untrack
spawn, kill
can have
User Mirror
+ gaze
represented by
Thing Mirror
is-a
represented by
Physical Thing

AW Concepts
Augmented Entities and Holograms
The full-fledge computational objects composing each AW
Conceptually located into the physical space
with a specific position, orientation and extension
defined in relation to the reference system of the AW
Designed with an observable state and a set of actions
agents and humans can observe entities and act over them
An AE exists in an AW even if it is not perceivable in terms of visual
impact
AWs allow to associates a geometry – an hologram – to each AEs
To provide a representation for the entity in the real world
Allows humans to see entities and interact whit them in Augmented
Reality meaning

AW Concepts
Agents
Cognitive Agents plays a key role in AW providing autonomous
behaviour and functionalities to physical/digital entities
When an agent join an AW, acquires the capabilities to create and
manage augmented entities (tracking them and perceiving observable
state)
In cognitive agents based on BDI model, belief about the state of the
AW and AEs are automatically managed, considering agents tracking
requirements

AW Concepts
Human Users
A human user starts a session inside an AW by means of an AW user
app
Is responsible to create an AR view, considering current position and
orientation of the user
Provides support to detect user inputs towards speciﬁc holograms
Many users can join the AW concurrently, each of them represented in
the AW by a User Mirror augmented entity
The state of a user must be made available to all interested entities,
using the User Mirror concept

AW Concepts
Physical World Coupling
An AE can oﬀers the medium to dynamically extend physical objects
functionalities
injecting MR beneﬁts into the IoT (and WoT) landscape
Bidirectional Augmentation
AEs must be coupled with related physical object
Changes in the physical object state tracked in the AE state and vice
versa.
Necessity of a“degree-of-freshness”to infer if and when the digital
state could be no longer aligned to the real state

MiRAgE
MiRAgE: a framework for Augmented Worlds
Mixed Reality based Augmented Environments
provides a runtime and an API to design, develop and deploy AWs
allows for the integration of agents and MAS in pervasive mixed
reality complex system
Key Features
1. Incremental augmentation degree
from ﬂat (smartphone-based) to full hologram-based MR experiences
2. Multi-user/cooperative systems
(multi)agents and (multi)humans cooperation/collaboration
3. Integrated with SW system engineering
support for heterogeneous mainstream technologies
integration of IoT/WoT stack and technologies

MiRAgE Architecture
MiRAgE Architecture

MiRAgE Architecture
MiRAgE Architecture
AW Runtime and WoAT
Hosts the execution of one or multiple instances of AWs
Manages the creation of AEs, the execution of their actions and allows
access to their observable properties
Proposes an interoperable interface to access to entities called WoAT
– Web of Augmented Things [CR17b, CR17a]
a RESTful interface based on HTTP operations and event-oriented
mechanisms
full interoperability and openness at application levelfor agents
technologies
based on (is a superset of) the Web of Things (WoT) model [GT16]

MiRAgE Architecture
MiRAgE Architecture
Hologram Engine
Is responsible of the management of holograms
geometries deﬁnition and updates
physics management
allows interactions in AR-style and input by human users (e.g.
gaze-based, hands/ﬁnger gestures, . . . )
. . .
Supports the AR/MR rendering of holograms
wraps AR/MR libraries and algorithms
enables marker-based and marker-less AR techniques

MiRAgE Architecture
MiRAgE Prototype Technologies
AW Runtime – mainly Java based
Vert.x library for implementing (part of) the WoAT layer
JSON used to model data exchanged in WoAT REST APIs calls
Hologram Engine – based on Unity 3D (and C# scripts )
Vuforia plugin to manage AR aspects
Mapbox plugin to manage GPS localization aspects
Augmented Entities – deﬁned as templates through Java-based API
exploiting ad-hoc annotations for types, ﬁelds and methods
Agents – designed and developed using the BDI model proposed
either by Jason/JaCaMo and ASTRA
CArtAgO and EIS libraries provided

MiRAgE Examples
The MiRAgE main beneﬁt
Designed with the goal to hide as much as possible implementation
details and enabling mechanisms of the AR/MR level
Idea: focusing only on the application logic!
1. Design AEs templates (deﬁning observable properties and actions of
each AE)
2. Design Hologram geometries (ore reuse provided ones)
3. Design application logic for agents living in the AW
4. Design application logic related to Humans interaction with the AW
5. Deploy and Run the AW

MiRAgE Examples
AE template example
Boat.java
@HOLOGRAM("Boat")
public class Boat extends AE {
@PROPERTY private double speedVal;
@PROPERTY private Vector2D speedVersor;
@PROPERTY private double windForce;
@ACTION
public void setSpeed(Vector2D versor , double val) {
customProperty ("speedVal", val);
customProperty (" speedVersor", versor);
}
@ACTION
public void setWindForce (double windForce) {
customProperty ("windForce", windForce);
}
}

MiRAgE Examples
Agents API
Primitive Actions Description
joinAW(name, location):
awID
to join an existing augmented world, getting an id
of the session
quitAW(awID) to quit from working in an augmented word
createAE(awID, name, tem-
plate, args, config): aeID
to create a new augmented entity in a specified aug-
mented world, specifying its name, template, param-
eters (that depend on the specific template), and
initial configuration (including position, orientation,
...)
disposeAE(aeID) to dispose an existing augmented entity
trackAE(aeID) to start tracking an existing augmented entity
stopTrackingAE(aeID) to stop tracking an existing augmented entity
moveAE(aeID, pos, orienta-
tion)
to change the position and orientation of an aug-
mented entity, if allowed
defineRegion(awID, name, re-
gion)
to define a named region, specifying a name and the
extension
trackRegion(awID, name) to start tracking a region
stopTrackingRegion(awID,
name)
to start tracking a region

MiRAgE Examples
Jason Agent examples
BoatPilot.asl
+! init
<- joinAW("museumAW"),
trackAE("boat"),
trackAE("wind"),
!navigate (30).
+! navigate(Speed)
<- getRndOrientation (Versor),
setSpeed(Versor , Speed),
.wait (5000) ,
!navigate(Speed).
@mng_border_collision [atomic]
+ borderReached
<- ! divertBoatRoute (180).
+windForce(V)
<- setWindForce (V).
LighthouseCtrl.asl
!init.
+! init
<- joinAW("museumAW"),
trackRegion ("museumAW","
red -zone").
+ regionUpdate ("red -zone", "
enter", AE)
<- turnOnLighthouse ,
registerRedZoneAccess (AE)
.
+ regionUpdate ("red -zone", "
exit", AE)
<- turnOffLighthouse ,
registerRedZoneExit (AE).

MiRAgE Examples
The Unity (customized) Editor

Case Studies and Applications
Case Studies
Augmented Museum and Augmented Harbour
Two real case studies in the cultural heritage ﬁeld (maritime
archaeology context)
with the collaboration of the“Museo della Regina”, Cattolica
Augmented Hospital
In the Healthcare 4.0 vision, (a part of) an hospital can be conceived
as an AW
Currently we are working with the purpose to build an Augmented
Emergency Room – the TraumaTracker Project
to support Trauma Team in managing emergencies
to assists Trauma Leader with a kind of Personal Assistant Agent
with the collaboration of the Trauma Center of the Cesena Hospital

Current State and Future Work
Current state and Considerations
AW is a conceptual framework for investigating and exploring the use
of cognitive agents and MAS for developing complex AR/MR systems
MiRAgE (with the WoAT layer) allows software designers and
developers to shape cyber-physical systems and smart environments
abstracting from specific enabling augmentation technologies
We promote an agent-oriented level of abstraction at the design level
Developed case studies have provided a first positive feedback about
the effectiveness of the approach

The AW Contribution to the State-of-the-Art
Shared/multi-user worlds
multiple human users immersed in the same AW
sharing and interacting with the same virtual entities
Bidirectional Augmentation
proposing a notion of augmentation wider than in MR pure context
about enriching physical environment with computational capabilities
Open Systems
where software agents and humans can dynamically join to a dynamic
augmented world
Enabling AR Technologies
allowing for a complete reuse of existing AR/MR enabling technologies
both software and hardware
Generality
supporting the development of AWs for diﬀerent application domains

The AW Contribution in Agents and MAS Research
The design of AWs is an interesting application domain for agents
research
challenging features
it’s a real (not simulated) testbed
Humans in the loop
The Humans-Agents Collaboration can be studied in situated contexts
A wide perspective for the research
Allow to simplify the design of Personal Agents for human assistance
in AWs, agents can“see what humans see”: both the real and the
virtual world elements

Future Work and Directions
A concrete evaluation of the AW conceptual model
through (ongoing) case studies and ad-hoc experiments
An improvement of the MiRAgE architecture and its implementation
to make it widely available as a reference platform to build
Agent-based Pervasive Mixed Reality Systems
Explore the application of AWs towards building Digital Twins
mainly, in the Industry 4.0
Explore agent design considering HCI techniques

References
References I
[CR17a] Angelo Croatti and Alessando Ricci, Towards the web of augmented things,
2017 IEEE International Conference on Software Architecture Workshops
(ICSAW), IEEE, April 2017, pp. 80–87.
[CR17b] Angelo Croatti and Alessandro Ricci, Mashing up the physical and
augmented reality: The web of augmented things idea, Proceedings of the
Eighth International Workshop on the Web of Things (New York, NY,
USA), WoT 2017, ACM, 2017, pp. 4–7.
[CSHO14] Abraham G. Campbell, John W. Staﬀord, Thomas Holz, and G. M. O’hare,
Why, when and how to use augmented reality agents (auras), Virtual Real.
18 (2014), no. 2, 139–159.
[GT16] Dominique Guinard and Vlad Trifa, Building the web of things: With
examples in node.js and raspberry pi, 1st ed., Manning Publications Co.,
Greenwich, CT, USA, 2016.

References
References II
[HCO+
11] Thomas Holz, Abraham G. Campbell, Gregory M. P. O’Hare, John W.
Stafford, Alan Martin, and Mauro Dragone, Mira-mixed reality agents, Int.
J. Hum.-Comput. Stud. 69 (2011), no. 4, 251–268.
[LA00] Michael Luck and Ruth Aylett, Applying artificial intelligence to virtual
reality: Intelligent virtual environments, Applied Artificial Intelligence 14
(2000), no. 1, 3–32.
[ORV08] A. Omicini, A. Ricci, and M. Viroli, Artifacts in the a&a meta-model for
multi-agent systems, Autonomous Agents and Multi-Agent Systems 17
(2008), no. 3, 432–456.
[Rou16] John Rousseau, The laws of mixed reality, or mixed reality without the
rose-colored glasses, 2016, https://www.artefactgroup.com/articles/mixed-
reality-without-rose-colored-glasses/.

References
References III
[RPLJ+
16] J. A. Rincon, J. L. Poza-Lujan, V. Julian, J. L. Posadas-Yague, and
C. Carrascosa, Extending mam5 meta-model and JaCalIVE framework to
integrate smart devices from real environments, PLOS ONE 11 (2016),
no. 2, 1–27.
[RPTC15] Alssandro Ricci, Michele Piunti, Luca Tummolini, and Cristiano
Castelfranchi, The mirror world: Preparing for mixed-reality living, IEEE
Pervasive Computing 14 (2015), no. 2, 60–63.
[RTP+
15] Alessandro Ricci, Luca Tummolini, Michele Piunti, Olivier Boissier, and
Cristiano Castelfranchi, Mirror worlds as agent societies situated in mixed
reality environments, Revised Selected Papers of the International
Workshops on Coordination, Organizations, Institutions, and Norms in
Agent Systems X - Volume 9372 (New York, NY, USA), Springer-Verlag
New York, Inc., 2015, pp. 197–212.

A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Similar to A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems

Similar to A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems (20)

Recently uploaded

Recently uploaded (20)

A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems