GhostHands is an application that takes advantage of the use of a virtual model of a mentor’s real hands, appearing in a workers full AR view of a work task. The mentor can see the workers view ‘through their eyes’ and can place their GhostHands into that view. Both worker and mentor appear to cope with this ‘ghost assisted’ experience very well, and recognise a sense of joint accomplishment from the cues that the hands embody in completing a task. Video demo: https://youtu.be/QZ5GdhSnMzQ

1. The GhostHands UX:
telementoring with hands-on
augmented reality instruction
Giuseppe Scavo
Knowledge Media Institute

2. 2
AR in manufacturing
2

3. 3
Advantages of AR & Telementoring
 Decreased error rate
 Decreased task completion time
 Enhanced awareness and safety
 Faster/cheaper training
3
“What happens during unforeseen situations?”
Is AR telementoring an
effective replacement for the
expert’s physical presence?
The help of the expert is often
crucial for the successful
completion of work/training
sessions that require specific
knowledge.

4. 4
AR telementoring vs. physical
presence
4
Communication
channel
AR tele-
mentoring
Physical
presence
Verbal Communication
Shared visual space
Overlaid information
Visual embodied learning
Manufacture: from latin “manu” (hand) + “factum” (made)

5. 5
GhostHands
5
GhostHands allows an instructor to
remotely stream his/her hands in an
“over-the-shoulder” way onto a worker’s
field of view.
Instructor and trainee share visual
space, audio channel and they also see
each other’s hands.
Imagine if you could see the hands of your instructor in
front you, interacting with the environment around you.

6. 6
GhostHands
6

7. 7
GhostHands: A new interaction
modality
Natural interface
Communication fastening
Conversation timing
Visual learning
Learn-by-doing experience
7

8. 8
Use cases
1-on-1 training sessions for new workers
Changes in the production process
Brand new machinery
New maintenance procedures
Machine faults
Safety critical situations
Any unforeseen scenario

9. 9
Implementation
9
Instructor-side application
Node.js server
Trainee-side application
Video stream
JSON tracking data

10. 10
UX evaluation framework
Human needs/expectations Product qualities
Relatedness/Conne
ctedness
Utility
Stimulation • Captivation
• Intuitiveness
• Playfulness &
Entertainment
Usability • Learnability
• Efficiency &
Effectiveness
• Safety/Trust
Competencies • Empowerment
• Increased awareness
and knowledge
• Efficiency &
Accomplishment
Visual
attractiveness
Security Hedonic
qualities
Autonomy
UX is often neglected in AR. However, it is extremely important to deliver
successful technologies.

11. 11
Experiment
 6 couples of participants, all researchers
 3 tasks of increasing difficulty to perform, designed after real-life
use cases:
 Rotor disc servicing. (19 steps)
 Motor/pump alignment. (24 steps)
 Thread feeder unit replacement. (33 steps)
 Worker equipped with head-worn camera. A display in background
showed the video feed with GhostHands superimposed
 Expert equipped with laptop and Leap Motion sensor. The expert
was briefed before the task.
 Post-task open answer questionnaires

12. 12
Findings
Naturalness & Intuitiveness
Frustration due to imprecision
“There were slight moments of frustration due to
losing the GhostHands tracking/motion”
Playfulness & Entertainment
Empowerment
“When struggling with or forgetting some name of
the component, you just point at it. The same
applies when you cannot find the proper verb or
description”

13. 13
Findings (2)
Connectedness & Joint
accomplishment
Efficiency
Autonomy
“GhostHands was as if I had a pair of expert
hands taking mine and guiding me through the
entire process”
“I think it can help in improving team-work
because it produces a sense of ‘shared
accomplishment’”
“I felt that sometimes the worker may rely
completely on the instructor being like a robot
executing the instruction without really learning”

14. 14
What’s next?
Deployment on smart glasses
Precision issue addressing
Quantitative analysis on error rate and task
completion time
Evaluation in real shop floor environment