1. TUG-KMI
Authoring System in TARGET
www.reachyourtarget.org
¨
Georg Ottl
Knowledge Management Institute
Cognitive Science Section
April 29, 2010
¨
Georg Ottl April 29, 2010 Page 1/37
2. TUG-KMI
Outline
Research environment
Competence performance assessment
Experts competence structure modeler
Probabilistic graphical models
Factor graphs
¨
Georg Ottl April 29, 2010 Page 2/37
3. TUG-KMI
Outline
Research environment
Competence performance assessment
Experts competence structure modeler
Probabilistic graphical models
Factor graphs
¨
Georg Ottl April 29, 2010 Page 3/37
4. TUG-KMI
Transformative, Adaptive, Responsive and enGaging
EnvironmenT (TARGET)
Serious game based learning
environment
Enterprise Competence
Development
Improve competences in the
project management and
innovation domain
¨
Georg Ottl April 29, 2010 Page 4/37
7. TUG-KMI
Role of TUG-KMI in TARGET
TUG-KMI responsible for TARGET learning process
TUG-KMI responsible for workpackage competence
development
Competence performance assessment component
Story adaptation/interventions
Integration competence development/TARGET learning
process
¨
Georg Ottl April 29, 2010 Page 7/37
11. TUG-KMI
TARGET competence performance assessment
Interpret observable performance in game experiences in
regards to a competence state1 .
Include motivational state emotional state in interpretation
Competence assessment as basis for macro and
microadaptive2 interventions and adaptations.
Computational model to automatically assess competence
state.
1
Klaus Korossy. “Modeling Knowledge as Competence and Performance”.
In: Knowledge Spaces: Theories, Empirical Research, Applications. Ed. by
Dietrich Albert and Josef Lukas. Mahwah, NJ: Lawrence Erlbaum Associates,
1999, pp. 103–132.
2
Dietrich Albert et al. “Microadaptivity within Complex Learning Situations
- a Personalized Approach based on Competence Structures and Problem
Spaces”. In: Proceedings of the international Conference on Computers in
Education (ICCE 2007). 2007.
¨
Georg Ottl April 29, 2010 Page 11/37
12. TUG-KMI
TARGET competence performance assessment model
authoring
Interpretation of the game experiences in terms of
competences can be done by a social community through
inspection.
Creation of a model by using the social community
observations input (cold start problem)
Experts create a model to automatically interpret performance
¨
Georg Ottl April 29, 2010 Page 12/37
13. TUG-KMI
TARGET competence performance assessment
requirements
Knowledge model/competence state exchange with HRM
Systems such as SAP.
Assessment in realtime3 to enable microadaptive interventions.
3
O. Conlan et al. Realtime Knowledge Space Skill Assessment for
Personalized Digital Educational Games. IEEE, 2009, pp. 538–542.
¨
Georg Ottl April 29, 2010 Page 13/37
14. TUG-KMI
Basic principle of probabilistic assessment of the
competence state
1. If the learner has the competence ci , than increase the
likelihood of all competence states γci containing ci and
decrease the likelihood of all competence states γ ci .
2. If the learner does not have the competence ci , than decrease
the likelihood of all competence states γci containing ci and
increase the likelihood of all competence states γ ci .
¨
Georg Ottl April 29, 2010 Page 14/37
15. TUG-KMI
Assessment calculation complexity reduction
No structure. Possibly 2n states to be updated on every
performance observation
Definition of a partial order relation on competences
exploiting the properties of the “PrerequesiteOf” relation type
reduces amount of possible competence states to be taken
into consideration.
Can experts or the community directly create a competence
assessment model??
Authoring tools can help to create a model for competence
assessment.
¨
Georg Ottl April 29, 2010 Page 15/37
16. TUG-KMI
Mathematical and computational model for competence
assessment
Nondeterministic assessment4
Traditional, multiplicative update rule56
Belief propagation networks such as Bayesian Networks7
4
C. Hockemeyer. “A Comparison of non-deterministic procedures for the
adaptive assessment of knowledge”. In: Psychologische Beitrage 44.4 (2002),
pp. 495–503.
5
Jean-Claude Falmagne and Jean-Paul Doignon. “A class of stochastic
procedures for the assessment of knowledge”. In: British Journal of
Mathematical and Statistical Psychology 41 (1988), pp. 1–23.
6
Jean-Claude Falmagne and Jean-Paul Doignon. “A markovian procedure
for assessing the state of a system”. In: Journal of Mathematical Psychology
32.3 (1988), pp. 232–258.
7
M. Villano. “Probabilistic Student Models: Bayesian Belief Networks and
Knowledge Space Theory”. In: Proceedings of the Second International
Conference on Intelligent Tutoring Systems. Springer, 1992, 491–498.
¨
Georg Ottl April 29, 2010 Page 16/37
17. TUG-KMI
Outline
Research environment
Competence performance assessment
Experts competence structure modeler
Probabilistic graphical models
Factor graphs
¨
Georg Ottl April 29, 2010 Page 17/37
18. TUG-KMI
Current state Competence Modeler
Support to create competence assessment models
Using the well studied PrerequesiteOf relation8
Support experts (psychologists) to create knowledge
structures.
8
Dietrich Albert et al. Knowledge Structures. Ed. by Dietrich Albert. New
York: Springer Verlag, 1994.
¨
Georg Ottl April 29, 2010 Page 18/37
20. TUG-KMI
Current state Competence Modeler
Problems solved and Related Problems
Computer supported Hasse diagramm creation
Visualizations done with the Java Universal Network/Graph
Framework (JUNG) framework9
9
J. Madadhain et al. “Analysis and visualization of network data using
JUNG”. In: Journal of Statistical Software 10 (2005), pp. 1–35.
¨
Georg Ottl April 29, 2010 Page 20/37
22. TUG-KMI
¨
Georg Ottl April 29, 2010 Page 22/37
23. TUG-KMI
Current state Competence Modeler
Problems solved and Related Problems
Creation of a Hasse diagram reduced to the problem of
calculating the minimal transitive reduction of a graph which
was shown to have the same complexity as calculation of the
transitive closure of a graph10 .
Effective calculation and detection of cycles by maintaining
the online topological order of the graph11
Visualizations done with the JUNG12
10
A. V. Aho, M. R. Garey, and J. D. Ullman. “The Transitive Reduction of a
Directed Graph”. In: SIAM Journal on Computing 1.2 (1972), pp. 131–137.
11
David J. Pearce and Paul H. J. Kelly. “A dynamic topological sort
algorithm for directed acyclic graphs”. In: J. Exp. Algorithmics 11 (2006),
p. 1.7.
12
J. Madadhain et al. “Analysis and visualization of network data using
JUNG”. In: Journal of Statistical Software 10 (2005), pp. 1–35.
¨
Georg Ottl April 29, 2010 Page 23/37
24. TUG-KMI
Current State Competence Modeler
On-Line Demo Afternoon
https://dev-css.tu-graz.ac.at/
¨
Georg Ottl April 29, 2010 Page 24/37
25. TUG-KMI
Outline
Research environment
Competence performance assessment
Experts competence structure modeler
Probabilistic graphical models
Factor graphs
¨
Georg Ottl April 29, 2010 Page 25/37
26. TUG-KMI
Probabilistic Graphical Models
Whatfor?
Simple way to visualize the structure of a probabilistic model
Graphical representation allows insights into the properties of
the model
Insights into conditional independence properties
Complex computations can be expressed in terms of graphical
representations; use of graph based inference algorithms that
exploit graph properties for calculation.
¨
Georg Ottl April 29, 2010 Page 26/37
27. TUG-KMI
Graph Terminology
A graph comprises vertices
V = (a, b, c, d) connected
by edges
Definition
A graph G is a pair G = (V , E ), where V is a (finite) set of
vertices and E ⊆ V × V is a (finite) set of edges.
¨
Georg Ottl April 29, 2010 Page 27/37
28. TUG-KMI
Definition
A graph G is called undirected iff
∀A, B ∈ V : (A, B) ∈ E ⇒ (B, A) ∈ E (1)
Two ordered pairs (A, B) and (B, A) are identified and represented
by only one undirected edge.
Definition
A graph G is called directed iff
∀A, B ∈ V : (A, B) ∈ E ⇒ (B, A) ∈ E (2)
An edge (A, B) considered to be a directed edge from A towards
B
¨
Georg Ottl April 29, 2010 Page 28/37
30. TUG-KMI
Probabilistic Graphical Models (1/2)
Whatfor?
In a probabilistic graph model every vertice represents a
random variable
The edges express probabilistic relationships between the
variables
Directed Graphical probabilistic Models
Bayesian Networks
Undirected Graphical Probabilistic Models
Markov Random Fields
Loose coupling between statistical variables.
¨
Georg Ottl April 29, 2010 Page 30/37
31. TUG-KMI
Graphical probabilistic models
Question: Can directed probabilistic models such as as
Bayesian networks be used for assessment. How does believe
propagation relate to the classical update rule?
Question: How is the relation between directed and
undirected probabilistic graphical models?
Use of directed and undirected graphical probabilistic models
to assess the players state
Efficient sum and dot product calculation.
¨
Georg Ottl April 29, 2010 Page 31/37
32. TUG-KMI
A flexible probabilistic graphical model, the Factor Graph
Factor Graphs13 as a single representation for directed and
undirected graphical probabilistic models
Factor Graphs were successfully applied for Bayesian Networks
and Markovian Models
Multiple applications in artificial intelligence and signal
processing based on Factor Graphs
13
Frank Kschischang et al. “Factor Graphs and the Sum-Product Algorithm”.
In: IEEE Transactions on Information Theory 47 (2001), pp. 498–519.
¨
Georg Ottl April 29, 2010 Page 32/37
33. TUG-KMI
Factor graph conversion (1/3)
Example 1:(A simple probabilistic graph) S1 S2
Let f (S1, S2, S3) be a function of three
variables, and suppose that f can be
expressed as a product
f (S1, S2, S3) = p(S1)p(S2)p(S3|S1, S2) S3
¨
Georg Ottl April 29, 2010 Page 33/37
34. TUG-KMI
Factor graph conversion (2/3)
S1 S2
Example 1:(A factor graph)
Let f (S1, S2, S3) be a function of three
variables, and suppose that f can be f
expressed as a product
f (S1, S2, S3) = p(S1)p(S2)p(S3|S1, S2)
S3
¨
Georg Ottl April 29, 2010 Page 34/37
35. TUG-KMI
Factor graph conversion (3/3)
Efficient algorithms available to calculate probabilities
(Sum-Product algorithm)
Makes extensive use of “conditional independent” properties
Parallelization possible
Approximative algorithms
Efficient marginalization
¨
Georg Ottl April 29, 2010 Page 35/37
37. TUG-KMI
[1] A. V. Aho, M. R. Garey, and J. D. Ullman. “The Transitive
Reduction of a Directed Graph”. In: SIAM Journal on
Computing 1.2 (1972), pp. 131–137.
[2] Dietrich Albert et al. Knowledge Structures. Ed. by
Dietrich Albert. New York: Springer Verlag, 1994.
[3] Dietrich Albert et al. “Microadaptivity within Complex
Learning Situations - a Personalized Approach based on
Competence Structures and Problem Spaces”. In:
Proceedings of the international Conference on Computers in
Education (ICCE 2007). 2007.
[4] O. Conlan et al. Realtime Knowledge Space Skill Assessment
for Personalized Digital Educational Games. IEEE, 2009,
pp. 538–542.
[5] Jean-Claude Falmagne and Jean-Paul Doignon. “A class of
stochastic procedures for the assessment of knowledge”. In:
¨
Georg Ottl April 29, 2010 Page 36/37
38. TUG-KMI
British Journal of Mathematical and Statistical Psychology
41 (1988), pp. 1–23.
[6] Jean-Claude Falmagne and Jean-Paul Doignon. “A
markovian procedure for assessing the state of a system”. In:
Journal of Mathematical Psychology 32.3 (1988),
pp. 232–258.
[7] C. Hockemeyer. “A Comparison of non-deterministic
procedures for the adaptive assessment of knowledge”. In:
Psychologische Beitrage 44.4 (2002), pp. 495–503.
[8] Klaus Korossy. “Modeling Knowledge as Competence and
Performance”. In: Knowledge Spaces: Theories, Empirical
Research, Applications. Ed. by Dietrich Albert and
Josef Lukas. Mahwah, NJ: Lawrence Erlbaum Associates,
1999, pp. 103–132.
¨
Georg Ottl April 29, 2010 Page 36/37
39. TUG-KMI
[9] Frank Kschischang et al. “Factor Graphs and the
Sum-Product Algorithm”. In: IEEE Transactions on
Information Theory 47 (2001), pp. 498–519.
[10] J. Madadhain et al. “Analysis and visualization of network
data using JUNG”. In: Journal of Statistical Software 10
(2005), pp. 1–35.
[11] David J. Pearce and Paul H. J. Kelly. “A dynamic
topological sort algorithm for directed acyclic graphs”. In: J.
Exp. Algorithmics 11 (2006), p. 1.7.
[12] M. Villano. “Probabilistic Student Models: Bayesian Belief
Networks and Knowledge Space Theory”. In: Proceedings of
the Second International Conference on Intelligent Tutoring
Systems. Springer, 1992, 491–498.
¨
Georg Ottl April 29, 2010 Page 37/37
40. TUG-KMI
Acronyms
JUNG Java Universal Network/Graph Framework
¨
Georg Ottl April 29, 2010 Page 37/37