Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Non-invasive data tracking in educational games
1. Knowledge Management Institute
Non-invasive data tracking in educational
games
Combination of Logfiles and Natural
Language Processing
Stephanie B. Linek, Georg Öttl, Dietrich Albert
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
1
2. Knowledge Management Institute
Outline
• Introduction
• Core factors of enjoyment in educational games
• Available data sources
• Open framework for a combined assessment
methodology
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
2
3. Knowledge Management Institute
Introduction
• Genre: adaptive educational games
• Transformative, Adaptive, Responsive and enGaging EnvironmenT
• Joy as essential factor in game based learning
• Do not destroy the overall game-play experience
• Address player’s needs
• Non-invasive user monitoring and data-tracking
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
3
4. Knowledge Management Institute
Core factors for enjoyment in games and
educational games
• Address the player’s need for reasonable parasocial
interaction
• Address the player’s individual preferences and
actual mood
• Address players capabilities
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
4
5. Knowledge Management Institute
Reasonable parasocial interaction
• Communication preferences are reflected by the
selected game genres
• Provide options to address different communication
preferences for a player
• Adaption to the communication preferences of the
player
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
5
6. Knowledge Management Institute
Individual preferences and actual mood
• Mood management
• Media experiences are used for regulation of one’s own mood [1][2]
• Mood monitoring
• Communication preferences provide evidence about mood
• Identification of player specific mood preferences
[1] Zillmann, D. (1988). Mood management through communication choices. American Behavioral Scientist, 31, 327-340.
[2] Zillmann, D. (2006). Dramaturgy for emotions from fictional narration. In J. Bryant & P. Vorderer (Eds.), Psychology of
entertainment (215-238). Mahwah, New Jersey: Lawrence Erlbaum Associates.
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
6
7. Knowledge Management Institute
Players capabilities
• Adapt to the players skills
• Adapt to the players progress during time
• Ongoing monitoring of the skills and cognitive
capabilities
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
7
8. Knowledge Management Institute
Data tracking in educational games
• Questionnaires
• Evaluation of the overall quality of the game
• Presented before and after the game
• Online behavior monitoring without interrupting the
ongoing learning process
• Logfiles
• Natural language
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
8
9. Knowledge Management Institute
Data tracking by logfiles
• Automatic protocol of actions in a computer based
environment
• Huge amount of behavioral information
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
9
10. Knowledge Management Institute
Defining associated logfile parameters
Logfile-Data / Probs (Potential) Meaning/interpretation
Mouse-clicking rate (number of clicks/time unit) Activity of the user: confusion/nervousness vs.
ambition
Frequency of tool-usage (of the different available Activity and direction/purpose of user-behavior
tools)
Frequency help-seeking behaviour Confusion / overload of the learner
Sequencing of material (in relation to the task) Approximation to the solution vs. disorientation
Interaction frequency/rate with NPC or another Affiliation/Intensity of parasocial interaction with an
player NPC / another player: positive (sympathy) vs
negative (aversion)
Length of Interaction: Time spent on interaction with Affiliation/Intensity of parasocial interaction with an
an NPC or another player NPC / another player: positive (sympathy) vs
negative (aversion)
Eye-contact between player’ avatar and the Affiliation/Intensity of parasocial interaction with an
NPC/another player’s avatar NPC / another player: positive (sympathy) vs
negative (aversion)
Number of characters (NPCs and/or other player’s Integration in the (virtual) community/group
avatars) the player is interacting with (Integration-Index)
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
10
11. Knowledge Management Institute
Pros and Cons of Logfiles
Pros Cons
•Objective measurement of •Too much data /
user behavior information overload
•Does not disrupt gameplay •Ambiguous data without
subjective meaning – hard
•Non reactive method to interpret
without demand effects
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
11
12. Knowledge Management Institute
Data tracking in Natural Language
• Natural language used in
• Team speak
• Multiparty chat conversations
• Private 1:1 chats
• Connection between the words an individual uses and mind
[3][4] [5]
• Attempt to algorithmically understand the subjective meaning of text with
Natural Language Processing (NLP)
[3] Jackendoff, R. S. (1985). Semantics and Cognition (Current Studies in Linguistics) (p. 304). The MIT
Press.
[4] Noam Chomsky. (1969). Aspects of the theory of syntax. MIT Press.
[5] Schank, R. C. (1972). Conceptual Dependency: A Theory of Natural Language Understanding. Cognitive
Psychology, 631(3), 552-631.
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
12
13. Knowledge Management Institute
Defining associated NLP parameters
• Identify locations, persons, organizations, …
• Identify the topic
• Identify fascinating, acceptable and unsuitable answers [6]
• Interpret the semantic orientation [7]
• Identify gender, deception status and culture [8]
[6] Huang, J., Zhou, M., & Yang, D. (2007). Extracting chatbot knowledge from online discussion forums. In
Proc. of IJCAI (pp. 423-428).
[7] Hatzivassiloglou, V., & McKeown, K. R. (1997). Predicting the semantic orientation of adjectives.
Proceedings of the 35th annual meeting on Association for Computational Linguistics - (pp. 174-181).
Morristown, NJ, USA: Association for Computational Linguistics.
[8] Chung, C. K., & Pennebaker, J. W. (2007). The psychological functions of function words. Social
communication: Frontiers of social psychology, 343-359.
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
13
14. Knowledge Management Institute
Pros and Cons of NLP
Pros Cons
•Works autonomous •Error prone
•Provides different •Ambiguous outcomes
qualitative information than •Mostly relies on supervised
logfiles machine learning
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
14
15. Knowledge Management Institute
Combination of logfile analysis and NLP
• Logfiles
• Quantitative objective data regarding the activity of the user
• NLP
• Qualitative information
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
15
16. Knowledge Management Institute
Multimodal classification using logfile and NLP
data
• Analyse common properties of both data sources
• Augment necessary additional information
• Combination of both
• Multimodal classification task
• Indicator selection based on empirical psychological studies
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
16
17. Knowledge Management Institute
Open framework for a combined assessment
methodology
Defining rules for multimodal
classification: Analysis of common
properties and augmenting information
Behavioural logfile
NLP indicators
parameter
Defining associations between logfile Defining associations between NLP
parameters and potential subjective generated information and potential
meaning subjective meaning
Pool of interesting psychological variables
subjective meaning
Theoretical and empirical research
as basis for the definition of appropriate indicators, variables, associations and rules
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
17
18. Knowledge Management Institute
Conclusion
• Multimodal classification of logfiles and NLP will lead
to a more holistic view of the user/player
• Future research on games and game-based learning
might lead to new insight on the association of user-
behaviour and subjective meaning
• A open framework in the sense that other additional
data sources could be incorporated and connected
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
18
19. Knowledge Management Institute
OUTLOOK
TARGET : www.reachyourtarget.org
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
19
20. Knowledge Management Institute
Thank You For Your Attention!
Questions?
Stephanie Linek: stephanie.linek@uni-graz.at
Georg Öttl: georg.oettl@tugraz.at
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
20
21. Knowledge Management Institute
References
[1] Zillmann, D. (1988). Mood management through communication choices. American
Behavioral Scientist, 31, 327-340.
[2] Zillmann, D. (2006). Dramaturgy for emotions from fictional narration. In J. Bryant & P.
Vorderer (Eds.), Psychology of entertainment (215-238). Mahwah, New Jersey:
Lawrence Erlbaum Associates.
[3] Jackendoff, R. S. (1985). Semantics and Cognition (Current Studies in Linguistics) (p.
304). The MIT Press.
[4] Noam Chomsky. (1969). Aspects of the theory of syntax. MIT Press.
[5] Schank, R. C. (1972). Conceptual Dependency: A Theory of Natural Language
Understanding. Cognitive Psychology, 631(3), 552-631.
[6] Huang, J., Zhou, M., & Yang, D. (2007). Extracting chatbot knowledge from online
discussion forums. In Proc. of IJCAI (pp. 423-428).
[7] Hatzivassiloglou, V., & McKeown, K. R. (1997). Predicting the semantic orientation of
adjectives. Proceedings of the 35th annual meeting on Association for Computational
Linguistics - (pp. 174-181). Morristown, NJ, USA: Association for Computational
Linguistics. doi: 10.3115/976909.979640.
[8] Chung, C. K., & Pennebaker, J. W. (2007). The psychological functions of function
words. Social communication: Frontiers of social psychology, 343-359.
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
21
22. Knowledge Management Institute
Defining the interesting psychological
variables
• Subjective psychological indicators
• Related to meaningful parasocial interaction
• Related to mood management
• Related to the adaptivity on the learner characteristics
• General Psychological variables
• Curiousness of the player
• Confusion
• Positive versus negative mood
• Cognitive resources
• Objective behavioural correlates have to be found
Georg Öttl Valencia, 08/03/2010 NON-INVASIVE DATA TRACKING IN EDUCATIONAL GAMES
22