This study explored using EEG signals to adaptively adjust the difficulty level of a virtual reality training task in real-time based on a participant's cognitive load. Participants completed a target selection task across increasing difficulty levels while EEG data was collected. Results showed reaction times remained consistent while brain activity, indicating cognitive effort, increased at higher levels. This suggests adaptive VR training can elevate cognitive load without harming performance. Future work will make tasks more complex and predictive of user abilities. Integrating EEG into VR headsets may improve such adaptive systems.

1. Exploration of an EEG-Based
Cognitively Adaptive Training
System in Virtual Reality
Arindam Dey | Alex Chatburn | Mark Billinghurst
University of Queensland, University of South Australia
Conference Paper

2. Virtual Reality Training
• Bluedrop Training and Simulation
• Helicopter hoist training
2

3. Adaptive Training
•Training content dynamically changes depending
on the learning state of the user
• Task performance
• Test performance
• Subjective measures
•None of these measure users mental activity or
cognitive load during the task

4. Mentor
● EEG + Desktop ITS learning system
● Teaching Reverse Polish Notation
● Maintain the learner in a positive mental state
● Select next learning activity that fits their current state
● using the workload and the engagement indexes
● understand = ask Q, confused = more examples
● Group with the mental state-based adaptive learning
● higher learning outcomes (23% higher score)
● better learning satisfaction (20% higher survey score)
Chaouachi, M., Jraidi, I., & Frasson, C. (2015, June). MENTOR: a physiologically controlled
tutoring system. In International Conference on User Modeling, Adaptation, and
Personalization (pp. 56-67). Springer, Cham

5. Motivation
Making VR training systems adaptive in
real-time to the trainee’s cognitive load
to induce best level of performance gain

6. Background
Gerry et al. [2018]
- Similar system
- Task complexity
not varied
- Limited analysis
Zhang et al. [2017]
- EEG for cognitive load
- VR driving simulator
- Autistic children
Friedman et al. [2007]
- EEG for locomotion
- CAVE system

7. Background Summary
● VR training tools are effective and skills
transferable to real world
● Earlier use of EEG in VR studies
○ Interaction
○ Measurement
○ Did not adapt
• We are first to explore real-time adaptive VR
training system using workload calculated from EEG

8. System
Oz, O1, O2, Pz, P3,
and P4

9. Adaption/Calibration
● Establish baseline (alpha power)
● Two sets of n(1, 2)-back tasks to calibrate own
task difficulty parameters
● Measured alpha activity (task load) and
calculated mean of the two tasks
● Mean → Baseline
● In experimental task
○ load > baseline → decrease level
○ load < baseline → increase level

10. Experimental Task
•Target selection
• number of objects, different colors
• shapes, and movement
Increasing levels (0 - 20)

11. Experimental Task
Difficulty - Low Difficulty - High

12. User Study
● Participants
● 14 subjects (6 women)
● 20 – 41 years old, 28 years average
● No experience with VR
● Measures
○ Response time
○ Brain activity (alpha power)
•5 minutes fixed trial time

13. Adaption
Time (s)
Baseline
300 200 100
Alpha
Power
3
4
5
6
87
9 1011 1213 1415
16
15
17
18
17
18
19
18
Baseline

14. Results – Response Time
Increasing levels
Response Time (sec.)
No difference between
easiest and hardest levels

15. Results – Time Frequency Representation
•Task Load
• Significant alpha synchronisation in the hardest difficulty levels
of the task when compared to the easiest difficulty levels
Easiest Hardest Difference

16. Key Finding
Similar reaction time but increased brain
activity increased cognitive effort in higher
levels to sustain performance

17. Limitations
•Task
• Should be similar to real-world tasks
•Behavior
• Difficulty levels could be designed differently
•EEG
• Only considered alpha activity ignoring theta
• Limited number of electrodes (6 on sides of head)

18. Conclusions/Future Work
• Conclusions
• Adaptive VR training can increase the user’s cognitive load
without affecting task performance
• First demo of the use of real-time EEG signals to adapt the
complexity of the training stimuli in a target acquisition context
•Future Work
• Significantly increase task complexity
• Can predict user performance based on the cognitive
capacity
• Using AR display
• See real world and more distractors

19. HMDs with Integrated EEG
● A number of HMDs available with integrated EEG
● Neurable – 6 electrodes in visual cortex
● Looxid - 6 electrodes frontal lobes
Neurable Looxid

20. Thank You
Arindam Dey
a.dey@uq.edu.au
University of Queensland
Fully Funded PhD Positions Available
Mark Billinghurst
mark.billinghurst@unisa.edu.au
University of South Australia
University of Auckland