Virtual and Augmented Reality Analytics Digility

Virtual and AugmentedVirtual and Augmented
Reality AnalyticsReality Analytics
Virtual and AugmentedVirtual and Augmented
Reality AnalyticsReality Analytics
Frog #84027132 © julien tromeur – fotolia.com - extended licenseFrog #84027132 © julien tromeur – fotolia.com - extended license
transformed into VR Frog by Joerg Osarektransformed into VR Frog by Joerg Osarek
Joerg OsarekJoerg Osarek
IT-avant-gardist, Business-IT-ArchitectIT-avant-gardist, Business-IT-Architect
Skilltower InstituteSkilltower Institute
Partner, its-peoplePartner, its-people
DigilityDigility
September 2016September 2016
Cologne, GermanyCologne, Germany

© Skilltower Institute, 20162
Intro: Joerg Osarek – vr.skilltower.com
 ManagementManagement
 IT-avant-gardeIT-avant-garde
 Virtualization (VMware)Virtualization (VMware)
 Oracle, LinuxOracle, Linux
 Virtual Reality AnalyticsVirtual Reality Analytics
 Cinematic VRCinematic VR
 VR BusinessVR Business
 Co-founder:Co-founder:
www.its-people.dewww.its-people.de

Reports from the future – my passion ever since
Always some years ahead of Mainstream (Years Ahead of Mainstream = YAM)
 1989: HP: Computer Viruses: 10 YAM
 1998: DOAG: VRML – The future of the Internet: 9 YAM (2ndLife 2007)
 2007: DOAG: How does SOA influence Business Intelligence?: 3 YAM
 2010: DOAG: Cloud Computing and the Technology-Monster: 2 YAM
 2011: DOAG: The Organic Computing Age: 9 YAM?
 2012: DOAG: SoMoLo (Social, Mobile, Local) / Big Data: 3 YAM
 2015: DOAG: Virtual Reality Analytics: 3-5 YAM?

Reports from the future – my passion ever since
1998: DOAG: VRML – The future
of the internet
Many thanks to Wolfgang Taschner
for his archived copy

VR/AR Analytics
How VR and AR change business intelligence and how you
can integrate VR/AR Analytics in your BI-systems and data warehouses
 I. Introduction into VR/AR today and tomorrow
 II. VR/AR Analytics: Your new business intelligence gold mine
 II.a Integrate VR/AR Sources into your Big Data Systems
 II.b Chances and Challenges with VR/AR Analytics
 III. The 3 waves – from VR to AR to USEMIR: What future awaits us?

I. Introduction into VR/AR today and tomorrow

Virtual Reality: The power of Immersion
VR puts tough guys under a spell unlike any other medium.
Reactions are ranging
from delight to pure horror
Try it yourself
or visit our VR Meetup in Frankfurt.

Display Technologie
 2015: Samsung Gear, Oculus DK2,2015: Samsung Gear, Oculus DK2,
Mobile VR, Google Cardboard, +++Mobile VR, Google Cardboard, +++
 2016: Oculus Rift Consumer Version2016: Oculus Rift Consumer Version
Playstation VR (Morpheus)Playstation VR (Morpheus)
HTC Vive, FOVE (Eye Tracking)HTC Vive, FOVE (Eye Tracking)
Microsoft Hololens für EntwicklerMicrosoft Hololens für Entwickler
What is happening at VR / MR? - VR is not just headsets!

Display Technology
FulldomesFulldomes
Leia HolodisplayLeia Holodisplay
Magic LeapMagic Leap

Display Technology Recording
FulldomesFulldomes
Google JumpGoogle Jump
+ Youtube VR+ Youtube VR
JauntVR NeoJauntVR Neo
Lightfield CamLightfield Cam
Lytro ImmergeLytro Immerge

Display Technology Recording
Interaction
FulldomesFulldomes
Google JumpGoogle Jump
+ Youtube VR+ Youtube VR
JauntVR NeoJauntVR Neo
Lightfield CamLightfield Cam
LeapMotionLeapMotion
Fairy LightsFairy Lights
HologrammHologramm
zum Anfassenzum Anfassen
VR SculptingVR Sculpting
Google SoliGoogle Soli
Radar ChipRadar Chip
Disney DISHDisney DISH
ILM X-LabsILM X-Labs
Lytro ImmergeLytro Immerge

What is happening next?
 Voice Mood Analysis (e.g. cogito App)
 Brain Wave interfaces (e.g. NeuroSky Neuro Headset)
 Nerve Induction? (e.g. direct projection into retina or nerves)
 Mark Zuckerberg: „we'll be able to send emotions to each other soon“*
* Article at theverge.com from Rich McCormick on June 30, 2015.* http://www.theverge.com/2015/6/30/8873819/mark-zuckerberg-facebook-sending-emotions-soon

Quelle: Woman and pan: combined of two public domain images from www.pixabay.com
 Mark Zuckerberg: „we'll be able to send emotions to each other soon“
… could look like this:

Quelle: Woman and pan: combined of two public domain images from www.pixabay.com , Quelle: Angry driver: Vladimir Mucibabic/123rf.com #4554701_l
 Mark Zuckerberg: „we'll be able to send emotions to each other soon“
… could look like this:
LIKE ?LIKE ?

II. VR/AR Analytics:
Your new business intelligence gold mine

VR Analytics comes in two flavours
a) Visualization of analytics data in VR

VR Analytics comes in two flavours
a) Visualization of analytics data in VR
We don't cover this today.
b) Analyze VR experiences in your DWH

Let's Play

II.a Integrate VR/AR Sources into your Big Data Systems

Level 1: Your Mission
BUILD ME THE PERFECT
VR ANALYTICS PLATFORM.
MUHAHAHA...

Level 2: Make or buy? - Status VR Analytics vendorsLevel 2: Make or buy? - Status VR Analytics vendors
 cognitiveVRcognitiveVR:: http://cognitivevr.co/http://cognitivevr.co/ Analytics platform and plugins for Unity and Unreal Game EnginesAnalytics platform and plugins for Unity and Unreal Game Engines
 retinadVR.comretinadVR.com (Unity Plugin) → joined(Unity Plugin) → joined rothenbergventures.comrothenbergventures.com
(investing in frontier tech startups)(investing in frontier tech startups)
 http://ghostline.xyz/http://ghostline.xyz/ VR Analytics tool PSVR, Vive, Oculus (pc based)VR Analytics tool PSVR, Vive, Oculus (pc based)
 http://www.viarbox.com/2016/05/02/virtual_reality_analytics/http://www.viarbox.com/2016/05/02/virtual_reality_analytics/ - end to end Platform- end to end Platform
 http://www.finwe.mobi/main/liveheat/http://www.finwe.mobi/main/liveheat/ - 360 video heat maps- 360 video heat maps
 Election.tvElection.tv Engages Kosher Virtual Reality Analytics For Exclusive 2016 US Election Tech PollingEngages Kosher Virtual Reality Analytics For Exclusive 2016 US Election Tech Polling
Partnership (Partnership (Kosher.tvKosher.tv, a media ventures company), a media ventures company)
 Some BI / DWH experts start to broaden their portfolioSome BI / DWH experts start to broaden their portfolio
(like(like its-peopleits-people andand skilltower instituteskilltower institute))
 It is quite early for the large BI / DWH software vendors but we should watchIt is quite early for the large BI / DWH software vendors but we should watch
trendsetterstrendsetters like:like: IBM, SAP, Qlik, Oracle, SAS, Information Builders,IBM, SAP, Qlik, Oracle, SAS, Information Builders,
MicrostrategyMicrostrategy andand challengerschallengers like:like: Microsoft, Tableau, Tibco, Infor, BoardMicrosoft, Tableau, Tibco, Infor, Board
and some more.and some more.

2USEMIR:
VR/AR Analytics
links oben:
www.retinad.io
Mitte:
cognitivevr.co
DemoDemo
timetime

CognitiveVR scene explorer in ordinary web browser

Robert Merki, CognitiveVR:Robert Merki, CognitiveVR:
We also provide users with aWe also provide users with a
multitude of pre-built analytics calls.multitude of pre-built analytics calls.
These include algorithms forThese include algorithms for
detecting comfort,detecting comfort,
controller/HMD occlsion,controller/HMD occlsion,
VR performance,VR performance,
and even arm length of users.and even arm length of users.
Robert Merki from cognitiveVR.co (Canada) holding my book – Selfie Image from Robert Sept.19th.2016

Robert Merki, CognitiveVR:Robert Merki, CognitiveVR:
The user who bought the gameThe user who bought the game
could not get passed a certain level.could not get passed a certain level.
Upon analysis of the session,Upon analysis of the session,
it was clear thatit was clear that this particularthis particular
user was too short to see the buttonuser was too short to see the button
that needed to be press[ed]that needed to be press[ed]
to open the next area!to open the next area!

Level 3: Approaches for displaying VR in headsets
Hybrid Rendering
 Nozon.com Presenz
 some light field tech
Live 3D in Game engine
 (Unity3D, Cryengine
Unreal Engine, ...).
Pre-rendered
 YouTube VR or 360°
 VR Player apps, ...

measurement data, e.g.:measurement data, e.g.:
Position 5,3,1Position 5,3,1
Rotation 23°left, 18°upRotation 23°left, 18°up
Speed of movementSpeed of movement
(eye position, parallax)(eye position, parallax)
......
Mapping:Mapping:
Position in VRPosition in VR
Objects watchedObjects watched
InteractionsInteractions
……
Merging with otherMerging with other
Data WarehouseData Warehouse
ContentContent
Enriched AnalysisEnriched Analysis
determinedetermine
meaningmeaning
MeasureMeasure
AnalyzeAnalyze
←← (Interactive live return path) ←(Interactive live return path) ←
Level 4: How to build a semanticLevel 4: How to build a semantic
Meta Layer for VR Analytics?Meta Layer for VR Analytics?

 depends on the particular VR/MR experience.depends on the particular VR/MR experience.
 time matters as objects move over time!time matters as objects move over time!
 meaning must be mapped manually or...meaning must be mapped manually or...
 (soon) can be mapped automatically by autonomous analysis(soon) can be mapped automatically by autonomous analysis
algorithms (Google/Facebook recognize objects, text andalgorithms (Google/Facebook recognize objects, text and
speech in images, videos, audio) e.g. Google Translatespeech in images, videos, audio) e.g. Google Translate
Augmented Reality or Youtube TranscriptionAugmented Reality or Youtube Transcription
Level 4: How to build a semanticLevel 4: How to build a semantic
Meta Layer for VR Analytics?Meta Layer for VR Analytics?

2USEMIR: Neural Talk 2 – free image analysis with artificial intelligence (AI)
https://github.com/karpathy/neuraltalk2https://github.com/karpathy/neuraltalk2
http://cs.stanford.edu/people/karpathy/neuraltalk2/demo.htmlhttp://cs.stanford.edu/people/karpathy/neuraltalk2/demo.html

Level 5: data interchange between VR/AR and our reality
SmartWatch:SmartWatch:
heart rate, ...heart rate, ...
voice- andvoice- and
mood analysismood analysis
facial expressionfacial expression
analysisanalysis
NLP*NLP*
eye movementseye movements
*NLP = Neuro Linguistic Programming:*NLP = Neuro Linguistic Programming:
an approach to communication, personal development, and psychotherapyan approach to communication, personal development, and psychotherapy
created by Richard Bandler and John Grinder in California, United States in the 1970s.created by Richard Bandler and John Grinder in California, United States in the 1970s.

II.b Chances and Challenges with VR/AR Analytics

Challenges of the emerging VR market:
Mistakes to avoid with VR/AR Analytics
 The even Bigger Data challenge
 technical fragmentation challenge
 The challenge of Standards and content vs. technology improvements
 The challenge of Data privacy and Security
 The challenge of patent wars

The even Bigger Data challenge
BigBig
DataData

The even Bigger Data challenge
BigBig
DataData
GiganDataGiganData
VR/ARVR/AR
turnturn
intointo

solution approaches: interpolate Data reduction
– example from Blender
– simplify F Curves addon.
For a VR Analytics system this
has to be done by an automated
algorithm.

Herausforderungen im VR Markt und Lösungsansätze
Challenges:
 Big Data → GiganData
 Technical Fragmentation
 Standards vs. Innovation
 Data Privacy and Security
 Patent Wars
Approaches for solutions
 interpolate, translate to meaning
 Include intermediate results layer and
abstraction tiers
 Include intermediate results layer
 open dialogue / co creation,
security concept for
VR/AR similar
to BYOD
 ...

Level 5: facial
expression analysis
Science Fiction?
Only for Headsets?
predicted in 2012 at DOAG for 2017,predicted in 2012 at DOAG for 2017,
already reality in 2015already reality in 2015

Level 5: facial
expression analysis
Emotion Analytics
with PC or smartphone
www.affectiva.comwww.affectiva.com
The video, that watches you.The video, that watches you.
analysis of facial expressions and your moodanalysis of facial expressions and your mood
with a simple webcam → Emotion Database with APIwith a simple webcam → Emotion Database with API

Level 5: facial
expression analysis
Yesterday:Yesterday:
What did the user do while interacting?What did the user do while interacting?
Today:Today:
What is the user doing while interacting?What is the user doing while interacting?
Tomorrow:Tomorrow:
What is the user going to doWhat is the user going to do while interacting?while interacting?
(interactive and responsive, blending worlds)(interactive and responsive, blending worlds)

Mission accomplishedMission accomplished
 measurement data → semantic layer
 combination with
additional data
(heart rate, facial expressions,
eye movement)
 combination with
classic data warehouse
data streams
 enables (NLP)
conclusions
close to thought reading and
prediction of behaviour
YES. YES.
MUHAHAHA...

III. The 3 waves –
from VR to AR to USEMIR: What future awaits us?*
*which has a huge impact on
our business intelligence systems
and our culture

Level up!Level up!
which will lead us to:which will lead us to:
USEMIRUSEMIR
UUbiquitiousbiquitious SESEnsorynsory MIMIxedxed RRealityeality
*Quantum*Quantum
TechTech
GeneticsGenetics RoboticsRobotics ITIT Nano-TechNano-Tech Space-TechSpace-Tech
accellerated by Q-GRINS*accellerated by Q-GRINS*

2USEMIR: Next generation of
mobile 3D devices
 Google builds VR Unit
+ Google Cardboard
+ Google Jump / Daydream
+ YouTube VR / Spotlight Stories
+ ca.1.4 Billion Investment in
MagicLeap 2014, 2015 Google + others
+ self driving cars
 Apple seems to build VR Unit
+ acquired primesens
+ acquired faceshift
+ hired Doug Bowman
+ plans for self driving cars
Bildquelle: Project Tango, Google.com
First Google Tango Phone:First Google Tango Phone:
Lenovo PHAB2 Pro announcedLenovo PHAB2 Pro announced
for September 2016 for 499,-€for September 2016 for 499,-€

2USEMIR: MSQRD (simple smartphone camera – real time)
erhältlich im iStore und Playstore: https://play.google.com/store/apps/details?id=me.msqrd.android&hl=de

2USEMIR: Real-Eyes lightfield display without headset
www.real-eyes.eu – own photos by Joerg Osarek.

11 Stereo-photos
with photogrammetry
turned back into
3D model.
2USEMIR: Real-Eyes lightfield display without headset
www.real-eyes.eu – own photos by Joerg Osarek.

USEMIRUSEMIR
tomorrow – or in the far future?tomorrow – or in the far future?
We will be surrounded by
U.SE.MIRU.SE.MIR
UUbiquitiousbiquitious SESEnsorynsory
MIMIxedxed RRealityeality
==
bidirectional intelligentbidirectional intelligent
interactioninteraction
only linear progressonly linear progress
necessary!necessary!
Images: presskit from Microsoft.com hololens, affectiva.com ,
presskit fromhttp://digitalnature.slis.tsukuba.ac.jp/2015/06/fairy-lights-in-femtoseconds/

I have an idea.
Let's provide them
with technology
to read our thoughts.
Quelle: Wikimedia.org – Erstellt 1590 – Public Domain
Who ought to know what about us? - a comparable encounterWho ought to know what about us? - a comparable encounter

What choices do we have?
back to the caves?back to the caves?
prepare to be terminated?prepare to be terminated?

What choices do we have?
 Wrong question!Wrong question!
Why do we fear a Terminator-Scenario?Why do we fear a Terminator-Scenario?
 Why don't we consider ourselves asWhy don't we consider ourselves as
self-confident honourable partners?self-confident honourable partners?
 Is there a third choice?Is there a third choice?

The third choice:
We can qualify for the
22nd century by
cultivating
chivalry and wisdom.
Q22=C+W
humbleness, honour,
kindness, confidence,
politeness, courage and bravery, clemency,
moderate behaviour ,
continuity, loyalty, dignity,
discipline and decency

The third choice:
We can qualify for the
22nd century by
cultivating
chivalry and wisdom.
Q22=C+W
self confidence
by acting
honourably
working together
as civilization
to boost our
development
EARNING BONUS POINTS:EARNING BONUS POINTS:
appearing as
self confident
and honourable
examples and
partners

LEVEL UP!LEVEL UP! INSERT COINSINSERT COINS
TO CONTINUETO CONTINUE
The rest is
up to us now.
We shape our future
together:
with our systems and
with our actions.
Remember:
"with great power
comes great
responsibility" (Stan Lee)

additional information
on VR/AR Analytics
read additionalread additional
informationinformation
in the book:in the book:

additional information
on VR/AR Analytics
3 video3 video
TrainingsTrainings
available soon atavailable soon at
training.vr.skilltower.comtraining.vr.skilltower.com

Contact and further information on Business VR / AR
Skilltower Institute
Joerg Osarek
Founder and CEO
Triftstr. 30
D-61350 Bad Homburg
Germany
mailto:joerg.osarek@skilltower.com
Tel.: +49(0)151-23 0 24 333
vr.skilltower.com

Virtual and Augmented Reality Analytics Digility

Recommended

Recommended

More Related Content

What's hot

What's hot (10)

Viewers also liked

Viewers also liked (20)

Similar to Virtual and Augmented Reality Analytics Digility

Similar to Virtual and Augmented Reality Analytics Digility (20)

Recently uploaded

Recently uploaded (20)

Virtual and Augmented Reality Analytics Digility