Presentation at the EURO mini conference on Logistics Analytics, held in Minsk, Belarus on June 18-19, 2018. Extended abstract: Industry 4.0 denotes increasing automation and data exchange in manufacturing, cyber-physical systems, the Industrial Internet of Things, cloud computing and cognitive computing. These technologies may be used to improve production quality through, for example, predicting malfunctions and responding in real time. Recent research reports that the digital technology can disrupt how and where activities are located and organized supply chains and which of the network actors inform about the value-adding transformations. Industry 4.0 technologies through tracking and monitoring of workers can transform the workplaces to be safer and more efficient. We propose that Augmented Reality (AR) can be counted among the Industry 4.0 developments. AR provides composite views to the user, enriching the perception of the ‘real’ world with virtual overlays in real time. It provides access to information that the users cannot directly detect with their own senses, which helps them to perform real-world tasks more efficiently. AR hardware is rapidly maturing with Google, Epson, Sony, Microsoft, to name just a few, are developing AR smart glasses (tens of millions are sold yearly), while AR-hardware-supported smartphones may be released in 2018. We predict that AR will be an integrating factor to enhance worker protection and reaching the goal of improving productivity. AR will take a key position in performance augmentation and workplace training needed for digitizing the industry.

1. 1 Molde University College, Norway
Mikhail Fominykh and Judith Molka-Danielsen
Augmented Reality
I n t e g r a t i n g H u m a n W o r k i n
A u t o m a t i o n a n d I n d u s t r y 4 . 0
Logistics Analytics conference
Minsk, Belarus | 18-19 June, 2018

2. 2 Molde University College, Norway
Outline
Augmented Reality in Industry 4.0
About Augmented Reality
Training and performance augmentation with
Augmented Reality
WEKIT Project
AR-FOR-EU Project

3. 3 Molde University College, Norway
Industry 4.0

4. 4 Molde University College, Norway
Industry 4.0

5. 5 Molde University College, Norway
Computerisation
Freya and Osborne (2016) The future of employment: How susceptible are jobs to computerisation?
https://doi.org/10.1016/j.techfore.2016.08.019

6. 6 Molde University College, Norway
Computerisation
Freya and Osborne “The future of employment: How susceptible are jobs to computerisation?”
https://doi.org/10.1016/j.techfore.2016.08.019
“Our model predicts that most workers in transportation
and logistics occupations, together with the bulk of office
and administrative support workers, and labour in
production occupations, are at risk.”
Freya and Osborne 2016

7. 7 Molde University College, Norway
Computerisation

8. 8 Molde University College, Norway
Augmented Reality in Industry 4.0

9. 9 Molde University College, Norway
Augmented Reality for a human in
Industry 4.0

10. 10 Molde University College, Norway
Wild, Perey, Helin, Davies and Ryan (2014) TELLME’s Learning Experience Model in
action on the shop floor of a textile weaving mill

11. 11 Molde University College, Norway
Scavo, Wild and Scott (2015) The GhostHands UX: telementoring with hands-on
augmented reality instruction, DOI: 10.3233/978-1-61499-530-2-236

12. 12 Molde University College, Norway
About Augmented Reality
Augmented Reality provides
composite views to the user, enriching
the perception of the tangible, ‘real’
world with digital ‘virtual’ overlays
visually and to all other senses.

13. 13 Molde University College, Norway
Industrial need
(CAUDELL & MIZELL, 1992)
BOING first use the term
‘AR’ in the beginning of
90s

14. 14 Molde University College, Norway
Academic research starting
• Motion stabilized display (Azuma, 1994)
• Fiducial tracking in video (Bajura & Neumann, 1995)
• UNC hybrid magnetic-vision tracker (State et al., 1996)

15. 15 Molde University College, Norway
ARTOOLKIT (1999)
First OPEN SOURCE
TRACKING LIBRARY

16. 16 Molde University College, Norway
MIT technology review 2007
AR is among 10 emerging technologies

17. 17 Molde University College, Norway
GOOGLE GLASS (2011)

18. 18 Molde University College, Norway
Error reduction in the Industry
(2014)
The use of Augmented Reality when
performing a 46-step task ranging in
complexity from selecting the correct
parts, to properly aligning and fastening
bolts through multiple parts reduced
time to task completion by 30% and
reduced errors (first time quality results
improved 90%).
RICHARDSON ET AL. (2014)
http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1090&conte
xt=imse_conf

19. 19 Molde University College, Norway
MICROSOFT HOLOLENS
(2016)

20. 20 Molde University College, Norway
Evolution of AR
1. Industry need
2. Academic research
3. Open Source
4. Mainstream
5. Industry use
6. What is next?

21. 21 Molde University College, Norway
Gartner Hype Cycle for Emerging Technologies, 2017

22. 22 Molde University College, Norway
Gartner Hype Cycle for Emerging Technologies, 2017

23. Maintenance
https://youtu.be/VGtCQWROytw?t=39s

24. Energy
https://youtu.be/lkH88TTgt6A?t=1m20s

25. Logistics
https://youtu.be/gnUK-HTn4ZA?t=41s

26. Picture copyright
Wearable Experience for
Knowledge Intensive Training -
WEKIT
Disclaimer
This project has received funding from the European Union’s
Horizon 2020 research and innovation programme under
grant agreement No 687669. http://wekit.eu/

27. Picture copyright

28. Why WEKIT.one
There are 26 million active enterprises with some 144
million persons employed in Europe alone. One third of
industrial enterprises in Europe offer continuing
vocational training.
Industries need high-quality specialized workplace
training if they want to stay competitive. Such training is
either ineffective or expensive. This makes expertise
transfer very difficult.
Holographic training and experience capturing enabled
by Wearables and Augmented Reality can address these
challenges by offering an effective training while
reducing the trainer’s workload.

29. WEKIT.one
Industrial Training Platform
WEKIT.one platform allows to create training scenarios in three steps
Capture
Experience
Select appropriate
Transfer Mechanisms
from the library and
demonstrate the tasks
and procedures by
performing them.
Re-enact
Experience
Perform the tasks and
procedures guided by
the captured
demonstrations and
assisted by formative
feedback.
Review and
Analyze
Review the recordings
in post-analysis,
compare performances,
discover parts where
improvement is
needed.

30. Capture Experience

31. Re-enact Experience

32. Review and Analyze
Experience

33. Compatible with the draft IEEE
ARLEM Standard
Augmented Reality Learning Experience Model
(ARLEM) is an integrated conceptual model and the
according data model specifications for representing
activities, learning context and environment.

34. 18/06/2018
WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE
TRAINING
34
Human
Performance
Augmentation
Smart Assistance: Intelligent
Training and Guidance Systems
Healthy workplaces: ergonomic,
safe, facilitating fitness
New work models

35. WEKIT Software
Trainer features
• Virtually annotating objects in the physical space (text, image, video, audio, 3d objects)
• Capturing expert performance using a multimodal sensor recording
• Transforming recording into ARLEM

36. WEKIT Software
Trainee features
• Importing ARLEM recording
• Automatically generated task list and guide functions to all annotations
• Contextualized multimodal guidance and re-enactment of experience
• Capturing trainee performance

37. WEKIT Software
Analysis features
• Mapping data to ARLEM
• Visualization of performance and biological data
• Comparison of multiple performances

38. Wearable design

39. Wearable Sensors and
Augmented Reality
Smart glasses
Smart glasses are worn on
the head, currently
Microsoft Hololens is used.
It functions both as one of
the sensors and as the
main augmenting device.
Posture
A combination of
accelerometers and
gyroscopes capture
posture movements of
the user in the
environment for real time
feedback and warnings.
Bio sensors
A number of sensors
collect biological data to
provide necessary
feedback and warnings in
real time or in post-
analysis, detecting such
states as stress, fatigue or
lack of focus.

40. Wearable Sensors and
Augmented Reality
Gestures
Hand gestures are
captured if they are
necessary to understand
the performance. Gestures
are also used to interact
with the system.
Force feedback
By applying silent
vibrational feedback on the
arms, the users can be
given guided feedback on
their actions.
Wearability
The hardware and sensors
are designed with
wearability and fashion in
mind.

41. Training Methodology
Research-based
Specifically for industrial training at the workplace
Designed for Augmented Reality and Wearables

42. Training Methodology
Step 1. Prepare
Break down complex tasks to subtasks, identify properties
of subtasks and
Select Transfer Mechanisms

43. Training Methodology
Step 2. Capture
Demonstrate each subtask while wearing the WEKIT
wearable solution
Step 3. Re-enact
Perform the tasks and procedures guided by the captured
demonstrations and assisted by formative feedback.
Step 4. Review and Analyze
Review the recordings in post-analysis, compare
performances, discover parts where improvement is
needed.

44. Aircraft Maintenance
Use Case

45. Medical Training
Use Case

46. Astronaut Training
Use Case

48. The Augmented Reality in Formal European University Education
AR-FOR-EU
p a r t n e r s h i p
Aim to establish and deepen a strategic partnership for teaching Augmented Reality in
Higher Education at scale on an undergraduate and graduate levels

49. AR-FOR-EU project has received
funding from the European Union’s
Erasmus Plus programme,
grant agreement
2017-1-NO01-KA203-034192.
Acknowledgement

53. 53 Molde University College, Norway
Q&A
mikhail.fominykh@himolde.no
http://www.mikhailfominykh.com/