Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Industrial Training and Workplace Experience with Augmented and Virtual Reality

Slides form the keynote at the Simposio Internacional de Informática Educativa (SIIE 2018)
http://siie2018.uca.es/index.php/en/keynotes-en/
Abstract: In the context of the 4th industrial revolution and a globalized world, there is a pressing need for continuous acquisition and update of skills to maintain efficiency and to ensure inclusion and participation of all citizens in the globalized workplace. At the highly automated and rapidly updated workplaces, the need for expertise and effective training is growing. In the EU-funded research-and-innovation project WEKIT, we address these challenges by developing a new approach to industrial training. This approach is based on the idea of using wearable sensors to capture expert performance and then making it available for trainees using Augmented Reality. The WEKIT training methodology and the technological platform allow creating effective educational experience efficiently using the time of the expert involved in content creation. The idea of capturing workplace experience finds another application area in the research project Virtual Internship, funded by the Norwegian welfare authority. In this project, we use augmented and virtual reality to increase awareness of schoolchildren about various professions and improve motivation of young unemployed to search for a new job. We aim to find out if immersive and interactive experiences of exploring workplaces and trying typical tasks can help in mitigating the youth unemployment.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Industrial Training and Workplace Experience with Augmented and Virtual Reality

  1. 1. Industrial Training and Workplace Experience with Augmented and Virtual Reality InternationalSymposiumonComputersinEducationSIIE2018 siie2018.adie.es University of Cádiz, Jerez Campus 19-21 September 2018 Mikhail Fominykh Researcher, Norwegian University of Science and Technology, Norway Project manager, Europlan UK ltd, United Kingdom
  2. 2. # 427/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING
  3. 3. # 527/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING
  4. 4. # 627/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING
  5. 5. Can Technology improve hands-on learning at the workplace? ? ???
  6. 6. Computerization Freya and Osborne (2016) The future of employment: How susceptible are jobs to computerisation? https://doi.org/10.1016/j.techfore.2016.08.019
  7. 7. Computerization
  8. 8. Industry 4.0
  9. 9. More and more complex workplaces have to be managed by less and less people
  10. 10. Workplace training will become more and more frequent
  11. 11. Augmented Reality
  12. 12. https://vimeo.com/133828847 Giuseppe Scavo, Fridolin Wild and Peter Scott (2015) The GhostHands UX: telementoring with hands-on augmented reality instruction, DOI: 10.3233/978-1-61499-530-2-236 Enriching the ‘real’ world with virtual visual overlays
  13. 13. vtt.fi Enriching the ‘real’ world with virtual visual overlays Kaj Helin, Timo Kuula, Jaakko Karjalainen, Iina Aaltonen, Fridolin Wild, Juhani Viitaniemi, Antti Väätänen (2015) Usability of the ARgh! Augmented Reality system for on-the-job learning See video at: https://www.youtube.com/watch?v=Snoyt5-pZRY
  14. 14. Augmented Reality for a human in Industry 4.0
  15. 15. Error reduction in the industry The use of Augmented Reality when performing a 46-step task ranging in complexity from selecting the correct parts, to properly aligning and fastening bolts through multiple parts reduced time to task completion by 30% and reduced errors (first time quality results improved 90%). 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 18 RICHARDSON ET AL. (2014) http://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1090&context=imse_conf
  16. 16. AR and wearables in Society Page 19WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING27/10/2018 o Wearables o Games o Tablets o AR glasses
  17. 17. AR and wearables in the Industry o Operations o Maintenance o Error prevention o Training Date Page 20
  18. 18. Wearable Experience for Knowledge Intensive Training - WEKIT Disclaimer This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 687669. http://wekit.eu/
  19. 19. WEKIT partners
  20. 20. Training with AR – knowledge conversion
  21. 21. Experience and knowledge Learning = converting experience to knowledge 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 24 Mikhail Fominykh, Fridolin Wild, Carl Smith, Victor Alvarez and Mikhail Morozov: "An Overview of Capturing Live Experience with Virtual and Augmented Reality”, DOI: 10.3233/978-1-61499-530-2-298.
  22. 22. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) separated 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 25
  23. 23. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 26
  24. 24. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 27
  25. 25. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 28
  26. 26. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 29
  27. 27. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 30
  28. 28. Experience and knowledge Learning = = converting experience to knowledge immediate experience (aka ‘practice’) information for a master level (aka ‘theory’) Experiencedlearner 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 31
  29. 29. Why WEKIT.one There are 26 million active enterprises with some 144 million persons employed in Europe alone. One third of industrial enterprises in Europe offer continuing vocational training. Industries need high-quality specialized workplace training if they want to stay competitive. Such training is either ineffective or expensive. This makes expertise transfer very difficult. Holographic training and experience capturing enabled by Wearables and Augmented Reality can address these challenges by offering an effective training while reducing the trainer’s workload.
  30. 30. Why WEKIT.one There are 26 million active enterprises with some 144 million persons employed in Europe alone. One third of industrial enterprises in Europe offer continuing vocational training. Industries need high-quality specialized workplace training if they want to stay competitive. Such training is either ineffective or expensive. This makes expertise transfer very difficult. Holographic training and experience capturing enabled by Wearables and Augmented Reality can address these challenges by offering an effective training while reducing the trainer’s workload.
  31. 31. Why WEKIT.one There are 26 million active enterprises with some 144 million persons employed in Europe alone. One third of industrial enterprises in Europe offer continuing vocational training. Industries need high-quality specialized workplace training if they want to stay competitive. Such training is either ineffective or expensive. This makes expertise transfer very difficult. Holographic training and experience capturing enabled by Wearables and Augmented Reality can address these challenges by offering an effective training while reducing the trainer’s workload.
  32. 32. WEKIT.one Industrial Training Platform WEKIT.one platform allows to create training scenarios in three steps Capture Experience Select appropriate Transfer Mechanisms from the library and demonstrate the tasks and procedures by performing them. Re-enact Experience Perform the tasks and procedures guided by the captured demonstrations and assisted by formative feedback. Review and Analyze Review the recordings in post-analysis, compare performances, discover parts where improvement is needed.
  33. 33. WEKIT methodology Capturing
  34. 34. Capture Experience
  35. 35. Capture Experience See video at: https://www.youtube.com/watch?v=AiKt9wJ3chg
  36. 36. WEKIT methodology Re-enactment
  37. 37. Re-enact Experience
  38. 38. WEKIT methodology Review
  39. 39. Review and Analyze Experience
  40. 40. WEKIT methodology Learning Experience Modeling
  41. 41. Compatible with the draft IEEE ARLEM Standard Augmented Reality Learning Experience Model (ARLEM) is an integrated conceptual model and the according data model specifications for representing activities, learning context and environment.
  42. 42. 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 46 Augmented Reality Learning Experience Model P15891 working group of the IEEE standards association http://arlem.cct.brookes.ac.uk
  43. 43. WEKIT Software Trainer features • Virtually annotating objects in the physical space (text, image, video, audio, 3d objects) • Capturing expert performance using a multimodal sensor recording • Transforming recording into ARLEM
  44. 44. WEKIT Software Trainee features • Importing ARLEM recording • Automatically generated task list and guide functions to all annotations • Contextualized multimodalguidance and re-enactment of experience • Capturing trainee performance
  45. 45. WEKIT Software Analysis features • Mapping data to ARLEM • Visualization of performance and biological data • Comparison of multiple performances
  46. 46. WEKIT Hardware
  47. 47. 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 51 Participatory design See video at: https://www.youtube.com/watch?v=ND913dF8iyc
  48. 48. Wearable prototype 1 Wearable prototype 2
  49. 49. Wearable prototype 3
  50. 50. Wearable Sensors and Augmented Reality Smart glasses Smart glasses are worn on the head, currently Microsoft Hololens is used. It functions both as one of the sensors and as the main augmenting device. Posture A combination of accelerometers and gyroscopes capture posture movements of the user in the environment for real time feedback and warnings. Bio sensors A number of sensors collect biological data to provide necessary feedback and warnings in real time or in post- analysis, detecting such states as stress, fatigue or lack of focus.
  51. 51. Wearable Sensors and Augmented Reality Gestures Hand gestures are captured if they are necessary to understand the performance. Gestures are also used to interact with the system. Force feedback By applying silent vibrational feedback on the arms, the users can be given guided feedback on their actions. Wearability The hardware and sensors are designed with wearability and fashion in mind.
  52. 52. Training Methodology Research-based Specifically for industrial training at the workplace Designed for Augmented Reality and Wearables
  53. 53. Training Methodology Step 1. Prepare Break down complex tasks to subtasks, identify properties of subtasks and Select Transfer Mechanisms
  54. 54. Training Methodology Step 2. Capture Demonstrate each subtask while wearing the WEKIT wearable solution Step 3. Re-enact Perform the tasks and procedures guided by the captured demonstrations and assisted by formative feedback. Step 4. Review and Analyze Review the recordings in post-analysis, compare performances, discover parts where improvement is needed.
  55. 55. Aircraft Maintenance Use Case
  56. 56. Medical Training Use Case
  57. 57. Astronaut Training Use Case
  58. 58. Evaluation results from trials 1 in 2017 have to be managed by Trials 2 are running now Summer-Autumn 2018
  59. 59. TAMARA Technology Acceptance Model for AR/WT (TAMARA) questionnaire contains some questions to check the technology acceptance of the user toward the proposed prototype. The questionnaire has been developed in Wild et al. (2017). The first 19 items use a 7-point Likert scale ranging from strongly disagree (1) to strongly agree (7), while the final item, usage frequency, is rated on a 6-point scale. They are presented to the experiment participants in the questionnaire in the following manner: ATU4 I look forward to those aspects of my job that require me to use AR & WT. 65 strongly disagree (1) disagree (2) somewhat disagree (3) neither agree or disagree (4) somewhat agree (5) Agree (6) strongly agree (7)
  60. 60. 66
  61. 61. SUS System Usability Scale (SUS) is a tool for measuring both usability and learnability. The SUS scores calculated from individual questionnaires represent the system usability. SUS yields a single number representing a composite measure of the overall usability of the system being studied. Scores for individual items are not meaningful on their own. SUS scores have a range of 0 to 100 (Brooke, 1996; 2013). According to validation studies, the acceptable SUS score is about 70 (Brooke, 2013; Bangor et al., 2009). SUS is based on John Brooke (2013): SUS: A Retrospective, In: Journal of Usability Studies, 8(2):29-40 67
  62. 62. SUS 68
  63. 63. SSQ The Simulator Sickness questionnaire (SSQ) was filled in by the participants before and after the test. The questionnaire before the test is used to make sure that the user does not have medical issues since the test has to be done only with healthy subjects. The questionnaire after the test is used to check the symptoms of the participants for three main areas: Nausea, Oculomotor and Disorientation. According to the NASA Johnson Space Centre Table, the score of the SSQ questionnaire are interpreted as: no symptoms if score < 0, negligible symptoms if score <5, minimal symptoms if score is between 5-10, significant symptoms if scores is between 10-15, concerning if score is between 15-20 and problem simulator if >20. 69
  64. 64. SSQ 70
  65. 65. SGUS Smart Glasses User Satisfaction (SGUS) questionnaire was created for the WEKIT trials. SGUS measures subjective satisfaction focusing especially on test participants’ experiences on the features that support learning. SGUS is based on the evaluation criteria for web-based learning by Ssemugabi & de Villiers (2007) and statements taken from Olsson, T. (2013) ‘Concepts and Subjective Measures for Evaluating User Experience of Mobile Augmented Reality Services’. SGUS consists of 11 items (statements) with a seven point Likert scale (1-7). For the analysis, the overall average from all items was calculated. Then this overall average was compared with averages from individual items on a scale of 1-7. 71
  66. 66. SGUS 72
  67. 67. QUIS The Questionnaire for User Interaction Satisfaction (QUIS) measures subjective satisfaction with specific aspects of the interface, including usability and user experience (Chin et al., 1988). It also measures the feeling about the AR glasses. QUIS was modified, using only the relevant items from the viewpoint of this study. Altogether 15 QUIS items with a scale mapped to numeric values of 1 to 7 were used. For the analysis, the overall average from all items was calculated. Then this overall average was compared with averages from individual items. 73
  68. 68. QUIS 74
  69. 69. TM The Transfer mechanism (TM) Questionnaire was designed to evaluate the adherence of the prototype to the Wekit framework. Each Transfer Mechanism was formulated into one or several self reflective statements Participants had no pre-knowledge of Transfer Mechanisms and were unaware that they were following the framework Both experts and students were asked to rate statements that were oriented to the successful implementation of each Transfer Mechanism after being exposed to the prototype. 75
  70. 70. TM 76 LT EB IT AL TE C
  71. 71. General conclusions • The organization and execution of the 3 trials was very successful • The use of questionnaires and interviews to quickly collect feedback from the users were a very good idea • The second iteration will implement the results and feedback collected during this first campaign of test • Particular emphasis in enhancing the design and the usability of the WEKIT applications • For both evaluation cycles we promised 600 people in total • We have achieved around 50 per Industrial case during iteration 1 • We had almost 150 people for Iteration 1; that leaves 450 people for Iteration 2, •Options (to be discussed): • 1) additional smaller trials • 2) more formal feedback gathered at conferences, workshops and fairs • 3) online surveys with videos and explanations (TAMARA, House of Quality) WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 7705/07/2017
  72. 72. 27/10/2018 78
  73. 73. Object- centric 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 80 Action- centric
  74. 74. Information communication 27/10/2018 WEARABLE EXPERIENCE FOR KNOWLEDGE-INTENSIVE TRAINING 81 Experience- communication Images: http://www.unmuseum.org/, http://en.wikipedia.org/, http://digitalxtrememedia.com/; https://www.youtube.com/watch?v=7d59O6cfaM0, https://www.microsoft.com/en-us/research/project/holoportation/
  75. 75. Virtual Internship in VR/AR with game elements Mikhail Fominykh Innovative Immersive Technologies for Learning, NTNU mikhail.fominykh@ntnu.no
  76. 76. Virtual Internship Project summary ►Simulation of workplaces with VR/AR ►Job taste: give an insight to professions ►Mastery/coping and security feeling about the typical job tasks ▼Mapping the user needs ▼Development of prototypes with Industry partners ▼Evaluation
  77. 77. Healthcare
  78. 78. Healthcare
  79. 79. Healthcare
  80. 80. FiskeoppdrettFish farming
  81. 81. Fish farming
  82. 82. Fish farming
  83. 83. Office work / startup
  84. 84. Main results of Evaluation in cycle 1 • Both councelors and young unemployed replied positively towards the concept of virtual internship. • The apps must be improved to reach the full potential. • The participants did not agree if gaming elements should be central or not, but collaboration and storytelling were rated positively • Many participants wished to have more workplaces. • There was no agreement among the participants if 360 videos or graphical simulations were best.
  85. 85. Figur 11: Jeg kan tenke meg å bruke appene ofte gjennom NAV. 1) Svært uenig, 2) Litt uenig, 3) Verken enig eller uenig, 4) Ganske enig, 5) Svært enig I would use such apps often at welfare centers Fully disagree Fully agree
  86. 86. Phase 2 Interview
  87. 87. Phase 2. Fishery VR See video at: https://youtu.be/Ry--5VdjTUA
  88. 88. Q & A Slides: https://www.slideshare.net/mfominykh

    Be the first to comment

    Login to see the comments

Slides form the keynote at the Simposio Internacional de Informática Educativa (SIIE 2018) http://siie2018.uca.es/index.php/en/keynotes-en/ Abstract: In the context of the 4th industrial revolution and a globalized world, there is a pressing need for continuous acquisition and update of skills to maintain efficiency and to ensure inclusion and participation of all citizens in the globalized workplace. At the highly automated and rapidly updated workplaces, the need for expertise and effective training is growing. In the EU-funded research-and-innovation project WEKIT, we address these challenges by developing a new approach to industrial training. This approach is based on the idea of using wearable sensors to capture expert performance and then making it available for trainees using Augmented Reality. The WEKIT training methodology and the technological platform allow creating effective educational experience efficiently using the time of the expert involved in content creation. The idea of capturing workplace experience finds another application area in the research project Virtual Internship, funded by the Norwegian welfare authority. In this project, we use augmented and virtual reality to increase awareness of schoolchildren about various professions and improve motivation of young unemployed to search for a new job. We aim to find out if immersive and interactive experiences of exploring workplaces and trying typical tasks can help in mitigating the youth unemployment.

Views

Total views

169

On Slideshare

0

From embeds

0

Number of embeds

4

Actions

Downloads

9

Shares

0

Comments

0

Likes

0

×