User-centric adaptation of serious video games continues to be a very important issue because of its benefits, such as enhanced motivation and engagement of individual players. It is based on player/learner characteristics that can be measured, estimated, recognized, or found by classification or clusterization. The paper suggests a new approach for dynamic, user-centric tailoring of task difficulty and the behavior of non-player characters. The approach is based on the emotional state and the shown outcomes of the individual player. Recognizing the current emotional state is based on facial expression analysis by convolutional neural networks and on an analysis of physiological data measured by sensors while playing the game. There are provided examples of serious games of learning with a dynamic adaptation of task difficulty and non-player character behavior.

1. Intelligent Adaptation of Difficulty and NPC Behavior
in Serious Video Games for Learning
Ivan Naydenov, Ilko Adamov, Boyan Bontchev
SUMMIT Annual Conference
Sofia, Bulgaria
April 24, 2024
Contract BG-RRP-2.004-0008 for the financing of project "Sofia University - Marking Momentum for Innovation and Technological Transfer“ under pillar 2 "Establishing a network of research higher education institutions in Bulgaria",
component "Innovative Bulgaria" from National Recovery and Resilience Plan part of the program to accelerate economic recovery and transformation through science and innovation

2. Two key issues of user-centric adaptation of
video games for learning
• Tailoring the learning task difficulty – it is very important because didactic
tasks are mapped to gaming tasks;
• Adapting the behavior of non-player characters (NPCs), i.e., virtual heroes –
applied in educational games for:
• (a) teaching assistants (the most popular case);
• (b) concurrent learners; or
• (c) opposite characters or enemies.

3. Recognizing emotions from facial expressions
• Understanding human behavior: recognizing emotions from facial expressions allows us
to gain insight into individual and collective emotional states.
• Communication and social interaction: facial expressions are a fundamental aspect of
nonverbal communication.
• User experience and human-computer interaction: recognizing emotions from facial
expressions is particularly relevant in the field of human-computer interaction – systems
and interfaces can be designed to adapt and respond accordingly.
• Psychological and mental health assessment: the ability to recognize and measure
emotions from facial expressions can be valuable in psychological and mental health
assessment.

4. Popular methods for emotion analysis
A. By using facial expressions:
• Classification using machine learning models like KNN, SVM, etc.
• Convolutional Neural Networks (CNNs)
B. Measurement and analysis of physiological data like Electrocardiograph
(ECG), Blood Volume Pulse (BVP), Galvanic Skin Response (GSR) and
Electromyography (EMG)
• Measure and extract physiological data
• Analyze and recognize emotions by data clustering

5. Recognition of player emotional state by CNN
• Colab (https://colab.research.google.com) with various Python libraries for data
analysis and visualization
• The neural network architecture consists of 779,718 neurons organized into 17
layers divided into four blocks.
• Each of the first three blocks contains three convolutional layers and one pooling
layer.
• Convolution layers employ filters or kernels to extract local features, such as edges,
corners, and textures, from the input image.
• Pooling layers are applied after convolution to reduce the spatial dimensions of the
feature maps using techniques like max pooling or average pooling.
• The final block includes a fully connected layer, which combines the high-level
features extracted from previous layers to produce the final classification output.

7. Training and validation accuracy (left) and loss
(right) of the neural network

8. Confusion matrix of
neural network

9. Samples of BVP (left) and GSR (right) data – used for data clusterization

10. K-Means (left), mean-shift (middle), and
agglomerative (right) clustering based on
physiological data

11. Adaptation of difficulty and NPC tutor behavior
The player:
• In situation 1 – has played a lot without any engagement and motivation;
• In situation 2 – has achieved high score and demonstrated good skills but remains in apathy;
• In situation 3 – is disappointed by his/her low outcome;
• In situation 4 – manages to play well but at the price of some non-desired emotions;
• In situation 5 – is happy to play the game but without trying to achieve good outcomes;
• In situation 6 – is happy to play the educational game while achieving a good score;
• In situation 7 – cannot achieve a good score and has both desired and non-desired emotions;
• In situation 8 – has succeeded in the game but at the price of negative emotions.
Situa-
tion
Desired
Emotion
Joy, Anger
Non-desired
emotion
Fear, Sadness
Out-
comes Difficulty
NPC
tutor
behavior
1 Low Low Low Const Encouraging
2 Low Low High Increase Satisfied, encouraging
3 Low High Low Decrease Soothing, encouraging
4 Low High High Decrease Encouraging
5 High Low Low Const Anger, surprised
6 High Low High Increase Satisfied
7 High High Low Const Encouraging, surprised
8 High High High Decrease Satisfied, surprised

12. Example 1: a car driving game
The first version of the game does not use any
adaptation methods.
The second version uses a classic method of
dynamic adaptation based on achieved levels of
player results. This changes the game's dynamics
and difficulty by altering environmental features
like fog, rain, darkness, and other factors.
The third version employs a dynamic adaptation
method that detects patterns in the player's
learning curve.

13. Example 2:
NPC’s with
simplified
representation
of emotional
state

14. Discussion
• Some players do not manifest their emotion by facial expressions while playing,
while others exaggerate their emotions in order to manipulate the game control
and to obtain a desired difficulty level by cheating the adaptation controller
• Measuring physiological signals by hardware devices and sensors is not
appropriate neither for online games (played at any place and time) nor for
playing desktop or console games in mass, even the sensors communicate
measured data in a wireless way.
• For user-centric adaptation, we could apply individual engagement, attention, or
motivation, provided we could measure or estimate these metrics.
• Measuring engagement and attention could be problematic, hence, classification
or clusterization approaches over physiological data can be applied here.

15. Conclusions
• The emotion-based adaptation approach is very general and could be
applied for tailoring other features of any serious video game [29] such as:
• Game mechanics – especially game rules, interface, and task automation;
• Informative feedback and help for each game task;
• Educational content;
• Audio-visual effects – such as both the tempo and volume of the music,
illuminations and shadowing, etc.
• This approach offers more engaging and tailored gameplay experiences
that cater to individual players' abilities and learning progress.

16. Thank you for your attention!
For questions – email to:
madvojd@gmail.com