V Jornadas eMadrid sobre “Educación Digital”. Vincent Aleven, Carnegie Mellon University: Supporting Self-Regulated Learningwith Intelligent Tutoring Systems

1July 2015 Supporting Self-Regulated Learning with Intelligent Tutoring Systems
Supporting Self-Regulated
Learning with Intelligent
Tutoring Systems
Vincent Aleven
Associate Professor
Human-Computer Interaction Institute
Pittsburgh Science of Learning Center (LearnLab)
Carnegie Mellon University
Based on the PhD research of
Yanjin Long

Introduction
• Intelligent tutoring systems (ITSs) can
support domain-level learning effectively
• How can we make them even more effective?
• How can an ITS help students improve as
self-regulated learners?
• Key aspects of self-regulated learning (SRL):
Self-assessment and problem selection
– Important in many learning environments!
• How can an ITS support self-assessment and
problem selection effectively? Do students
learn better as a result?

Overview
• Cognitive Tutors
• Role of self-assessment and problem selection
in self-regulated learning (SRL)
• 2 classroom studies of how to support these
SRL aspects in ITSs
• Discussion
Work by John Anderson, Ken Koedinger,
Albert Corbett, Steve Ritter, and others

Algebra Cognitive Tutor
Use graphs, graphics calculator
Analyze real world
problem scenarios
Use table,
spreadsheet
Use equations,
symbolic calculator
Tracked by
knowledge tracing
Model tracing to provide
context-sensitive Instruction

Real-world Impact of
Cognitive Tutor Courses
• Spin-off company Carnegie Learning, Inc.
• Over 500,000 students per year

Effectiveness of Cognitive Tutor
Algebra at Scale
• Over 17,000 students in
147 schools across 7 states
• Duration: 2 years
• Cost: $6 million
• Test: Algebra Proficiency
Exam, 32-item multiple-
choice assessment
• Research participants using
Cognitive Tutor Algebra
improved eight percentile
points, compared to the
control group
– About the same as doubling
math learning in a year for a
high-school student

• Cognitive Model: A system that can solve problems in
the various ways students can
Strategy 1: IF the goal is to solve a(bx+c) = d
THEN rewrite this as abx + ac = d
Strategy 2: IF the goal is to solve a(bx+c) = d
THEN rewrite this as bx + c = d/a
Misconception: IF the goal is to solve a(bx+c) = d
THEN rewrite this as abx + c = d
Cognitive Tutor Technology:
Use ACT-R theory to individualize instruction

3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3 6x - 5 = 9
If goal is solve a(bx+c) = d
Then rewrite as abx + ac = d
Then rewrite as abx + c = d
Then rewrite as bx+c = d/a
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction

3(2x - 5) = 9
6x - 15 = 9 2x - 5 = 3
Then rewrite as abx + ac = d
• Model Tracing: Follows student through their individual
approach to a problem -> context-sensitive instruction
Hint message: “Distribute a
across the parentheses.”
Bug message: “You need to
multiply c by a also.”
• Knowledge Tracing: Assesses student's knowledge
growth -> individualized activity selection and pacing
Known? = 85% chance Known? = 45%
6x - 5 = 9
Then rewrite as abx + c = d

Bayesian Knowledge Tracing
drives Open Learner Models
(OLMs) and Task Selection
(Cognitive Mastery)

Assumptions Underlying
Bayesian Knowledge Tracing
Learning assumptions
• Each rule is either learned or unlearned
• In problem-solving a rule can transition from
unlearned to learned at each opportunity to apply
it
• No forgetting - Rules do not transition back from
learned to unlearned
Performance assumptions
• If the rule is in the learned state there is some
chance the student will slip and make a mistake.
• If the rule is in the unlearned state there is some
chance the student will guess correctly.
(Corbett & Anderson, 1995; Corbett et al., 2000)

Inferring Learning State
• Following each opportunity to apply a rule, the new
probability estimate that the rule has been learned,
p(LN|EN), is:
p(LN|EN) = p(LN-1|EN) + (1 - p(LN-1|EN))*p(T)
Bayes Theorem
p(Ln-1|Evidencen) expands Bayes’ Theorem
p(Ln-1|Cn) = p(Ln-1)*(1-p(S))
p(Ln-1)*(1-p(S)) + p(Un-1)*p(G)
p(Ln-1|Incn) = p(Ln-1)*p(S)
p(Ln-1)*p(S) + p(Un-1)*(1-p(G))
(Corbett & Anderson, 1995; Corbett et al., 2000)

Self-Regulated Learning:
Great Theoretical Diversity

Background: Self-Regulated
Learning
• How do instructional intervention aimed at
supporting these elements affect robust
learning?
Planning
• Goal Setting
• Study Choice
Monitoring
and Control
• Self-Assessment
• Help Seeking
Evaluating
• Self-Explanation

Why is Self-Assessment
Important?
• The process of self-assessing can facilitate
deep thinking and reflection
(Boud, 2004; White & Frederiksen, 1998)
• The results of self-assessment can lead to
better learning plans and study choices, as
well as better learning outcomes
(Thiede, Anderson & Therriault, 2003; Winne &
Hadwin, 1998)
• However, students’ self-assessment is often
inaccurate
(Dunlosky & Lipko, 2007; Nelson, 1996)

Research Questions – Study 1
1. (How) can the Open Learner Model be
leveraged to support student self-
assessment in ITSs?
2. Does the self-assessment support lead to
more accurate self-assessment and better
learning outcomes?
3. Does study choice in an ITS lead to better
learning outcomes, especially when
combined with self-assessment support?

Geometry Cognitive Tutor
with Skill Meter

Skill Diary, Part 1

Skill Diary, Part 2

Skill Diary, Part 3

Study 1
• Hypothesis: Periodically filling out structured
Skill Diaries helps students self-assess and
learn better
• Participants:
– 122 students from 2 teachers’ 6 classes in a local
high school
– Complete data for 95 students
• Procedure: Students worked on tutor for 3
class periods (volume and surface areas for
spheres and right prisms), took paper pre-test
before and post-test after
• Experimental condition: Skill Diary
• Control condition: Control Diary (no self-
assessment)

Control Diary

Post-Test: Experimental Group
Better on Reproduction Problems
Mean Test Scores (SD)
Pre-Test
Reproduction
Post-Test
Reproduction
Pre-Test
Transfer
Post-Test
Transfer
Exp. Group 0.55 (.34) 0.62 (.29) 0.50 (.28) 0.58 (.26)
Ctrl. Group 0.46 (.44) 0.49 (.33) 0.46 (.22) 0.57 (.24)
F(1, 93) = 3.86, p = .052, η² = .040
Caveat: when pre-test score is used as co-variate, the difference
between two groups on reproduction problems was on the
borderline of significance (F(1, 92) = 2.75, p = .101, η² = .029)

Post-test: Lower Performing
Students Who Used Skill Diaries
Did Better
Test Scores on Reproduction Problems (SD)
Pre-Test Post-Test
Exp Ctrl Exp Ctrl
Lower-Performing
Group
0.35 (.45) 0.16 (.35) 0.53 (.47) 0.30 (.39)
Higher-Performing
Group
0.74 (.41) 0.74 (.75) 0.71 (.38) 0.68 (.41)
(F(1, 44) = 4.586, p = .038, η² = .094; pre-test reproduction problem
score was used as co-variate)

Measuring Self-assessment
Accuracy on Pre- and Post-Tests
• Measures the discrepancy between self-
assessed and actual performance.
(Schraw, 2009)

Self-Assessment Accuracy
Absolute Accuracy Index
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Pre-Test Post-Test
Lower-Performing
Higher-Performing
• Higher performing students have more accurate
self-assessment
Note: Lower means
more accurate

Self-Assessment Accuracy of
Lower-Performing Students
Absolute Accuracy Index
t(23) = 2.257, p = .034
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
SA on Pre-
Test
SA on Post-
Test
Experimental
Group
Control Group
• Accuracy of SA improves from Pre to Post for lower-
performing students

Used DataShop to Analyze Learning
Processes (i.e., ITS Analytics)
28
https://pslcdatashop.web.cmu.edu/

Process Measures
Correlations
Pre-
Test
Post-
Test
Number of Hints -.56** -.47**
Time Spent on Each Hint .20 .34**
Number of Incorrect
Attempts
-.35** -.32**
Assistance Score -.52** -.47**
Time Spent on Each Step -.19 -.20
* p <.05
** p <.01

Process Measures
Correlations Condition
Differences
Pre-
Test
Post-
Test
Exp Ctrl η²
Number of Hints -.56** -.47** .054 .082 .049*
Time Spent on Each Hint .20 .34** 17.5 12.4 .037*
Number of Incorrect
Attempts
-.35** -.32** .085 .092 .031
Assistance Score -.52** -.47** .140 .174 .055*
Time Spent on Each Step -.19 -.20 15.4 14.4 .027
* p <.05
** p <.01

Summary of Contributions
• Skill Diaries practical way of supporting
effective self-assessment for lower-
performing students
• Demonstrates a beneficial role of self-
assessment in students’ learning of
problem-solving tasks with an ITS

Research Questions
assessment in ITSs?
learning outcomes?

2 x 2 Experiment
• Two factors:
– OLM vs. no OLM
– Student control over problem selection vs.
system control (PS vs. noPS)
• Pre/post test before/after with
procedural and conceptual items
• Participants:
- 62 students from one teacher’s three 7th-
grade classes at a middle school in
Pittsburgh
- 56 students completed all five levels

Used Tutor Built with our ITS Authoring Tools
http://ctat.pact.cs.cmu.edu/

The linear equation tutor
(Waalkens, Aleven & Taatgen, 2013)

Use of Open Learner Model (OLM)
to Support Self-Assessment
Students are prompted to self-
assess …
At the end of each problem:
and then see the system’s
update of the skill bars

Use of Open Learner Model (OLM) to
Support Study Choice

Students Improved Significantly
from Pre-Test to Post-Test

OLM Conditions Did Significantly
Better on Post-tests
Main effect of OLM on the overall test score: p = .031, η² = .078
Main effect of OLM on the conceptual items: p = .026, η² = .082

Process Measures
Main effect of OLM on incorrect attempts (p = .062, η² = .059), on
assistance score (p = .009, η² = .116);
Main effect of PS on the assistance score (p = .075, η² = .056)

Do Self-Assessment Scores
Relate to Actual Test Scores?
Different way of asking if self-assessment was accurate

Research Questions
assessment in ITSs?
learning outcomes?

Students have moderate to high
accuracy of self-assessment
Absolute self-assessment (SA) accuracy for each condition
[Schraw, 2009]
A lower the index means more accurate self-assessment.

Summary of the Main Results
• All students improved significantly on
solving linear equations after using
the tutor
• The students who had access to the
OLM did significantly better on post-
test
• The students had moderate to high
accuracy of self-assessment

Contributions
• First controlled classroom experiment that
shows an OLM significantly enhances
students’ learning outcomes with an ITS
• Helps establish the important role of self-
assessment in students’ learning of problem
solving tasks

Final Thoughts
• Studies show that ITS technology can
be extended to support self-regulated
learning (SRL) skills
• Future work
– Study wider range of SRL processes and
learning environments
– Study influence on future learning: Did
students become better learners?
– Study SRL in the context of MOOCs

Year 9: Self-assessment and
study choice
1. How can we design the OLM so it
effectively supports self-assessment in
ITSs?
learning outcomes?

Year 10: full cycle of SRL
• Include other SRL processes, such as
goal setting
• Further redesign the Linear Equation
Tutor to support the SRL processes and
use it as the experiment platform for
classroom evaluations

Year 10: Shared OLM
• Open students’ progress information to
their peers through shared OLMs
• Bull, Mabbott and Abu-Issa’s (2007)
survey study pointed out the potential of
shared OLMs for fostering motivation and
supporting goal setting
• Few empirical studies have been conducted
to investigate the effects of shared OLMs
on supporting different SRL processes in
ITSs, as well as the further effects on
students’ learning outcomes and
motivation

Year 10: Shared OLM
• Use HCI methods to investigate how to best
design the shared OLM in the linear equation
tutor to allow students exchange and discuss
their goals, progress, specific errors made in
the tutor, understanding of their own learner
models, etc. with their peers
• Vary the SRL processes supported by the tutor
(mainly through the shared OLM) and conduct
a controlled experiment to investigate the
effects with and without the shared OLM

An ITS Success Case
• Most widely used ITS
– 2,700 schools across the country
– Marketed by spin-off company Carnegie
Learning
– Bought by the Apollo Group (runs University of
Phoenix)
• “Exemplary Curriculum” by US Dept of Ed
• Field studies reported in highly cited paper
– Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997).
Intelligent tutoring goes to school in the big city. International Journal of Artificial
Intelligence in Education, 8(1), 30-43.
. . .

• Tells us that cognitive tutor curriculum
is more effective than other curricula,
but tells us very little (if anything)
about why
• Useful for decision makers, not for
scientists …

Examples of Successful
Metacognitive Interventions
• Reading comprehension through “reciprocal
teaching” (Palincsar & Brown, 1984)
• Self-assessment of science inquiry cycle
(White & Frederiksen, 1998)
• Self-addressed metacognitive
questions(Mevarech & Fridkin, 2006)
• Reflecting on quiz feedback (Zimmerman &
Moylan, 2009)

• Geometry example?

Our prior work: survey and
interview on the use of the
Skillometer
• 44 experienced Cognitive Tutor users
• Students inspect the Skillometer frequently
during learning
• Students don’t actively self-assess or reflect
using the Skillometer
• So, not likely that simply presenting the
Skillometer supports students’ self-assessment
– E.g., improving its frequency or accuracy
[Long & Aleven, 2011]

Was the Skill Diary more effective
for lower-performing students?
• Previous literature suggested that good
students tend to have better self-assessment
accuracy [Chi et al, 1989]
• found their intervention was more helpful for
lower ability students (Hartley & Mitrovic,
2002)

Prior work on self-
assessment and OLM
• Hartley and Mitrovic (2002)
- Compared students’ learning gains when with or
without
access to an inspectable OLM
- Found no significant differences between conditions
• Long & Aleven (2011)
- Students inspect the OLM frequently
- Students don’t actively self-assess or reflect using
the OLM

View-1 of the new OLM on the
problem solving screen

View-1 of the new OLM on the problem
solving screen

solving screen

View-2 of the new OLM

noOLM+PS condition

OLM+noPS

noOLM+noPS

Prior Work on Supporting
Self-Assessment in ITSs
• Tutor that guides students through self-
assessment activities improved self-
assessment on better mastered skills
(Roll et al., 2011)
• Tutor that provides metacognitive prompts
and feedback improved students’ self-
assessment accuracy and learning efficiency
(Feyzi-Behnagh, Khezri, & Azevedo, 2011)
• Does self-assessment support lead to better
learning at the domain level?

Implications of the Log Data
Analysis
How does the self-assessment support influence
students’ learning behaviors?
1. More deliberate use of tutor help
2. More careful execution of the learning task (fewer
incorrect attempts)
3. More efficient learning

Replicated Field Studies
• Controlled, full year classroom experiments
• Replicated over 3 years in urban schools
• In Pittsburgh
& Milwaukee
• Results:
50-100% better on
problem solving &
representation use.
15-25% better on
standardized tests.
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997).
Intelligent tutoring goes to school in the big city. International Journal of
Artificial Intelligence in Education, 8(1), 30-43.
0
10
20
30
40
50
60
Iowa SAT subset Problem
Solving
Represent-
ations
Traditional Algebra Course

What is Metacognition?
• Reasoning about one’s own thinking, learning,
memory, etc.
(Brown, 1983)
• “Metacognitive learning strategies” are specific
kinds/uses of metacognition that aid learning,
including planning, checking, monitoring,
selecting, evaluating, and revising
(Schoenfeld, 1987)
• Key components of theories of self-regulated
learning (SRL)
(Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman,
2008)

Objectives in Supporting Metacognition
Improve future
domain learning
Improve current domain
learning in the supported
environment
Improve metacognitive strategies
in the supported environment
Improve future
metacognitive strategies
After the
metacognitive
intervention
During the
metacognitive
interventionIdo Roll

Background
• Broaden scope of SRL aspects studied
• Support self-assessment in the software
Planning
• Goal Setting
• Study Choice
Monitoring
and Control
• Self-Assessment
• Help Seeking
Evaluating

Why is (Student-Controlled)
Problem Selection Important?
• Computer-selected problem sequences led to
better learning than student-selected
(Atkinson,1972)
• Less able students learned with an ITS condition
that gradually increased student control
(Mitrovic & Martin, 2003)
• Adaptive navigation support in QuizGuide led to
increased students’ participation and better final
academic performance
(Brusilovsky et al., 2004)

Research Questions
assessment in ITSs?
learning outcomes?

Limitation and Future Work
• Small sample size of the experiment
• The effect of study choice on learning
outcomes needs further investigation
• Support other SRL processes in ITSs

What is an “Intelligent
Tutoring System” (ITS)?
• A kind of educational software
– Supports “learning by doing” with personalized,
step-by-step guidance
• Uses cognitive modeling and artificial
intelligence techniques to
– Provide human tutor-like behavior
– Be flexible, diagnostic & adaptive
– Provide personalized instruction (e.g., select
problems on an individual basis)

V Jornadas eMadrid sobre “Educación Digital”. Vincent Aleven, Carnegie Mellon University: Supporting Self-Regulated Learningwith Intelligent Tutoring Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to V Jornadas eMadrid sobre “Educación Digital”. Vincent Aleven, Carnegie Mellon University: Supporting Self-Regulated Learningwith Intelligent Tutoring Systems

Similar to V Jornadas eMadrid sobre “Educación Digital”. Vincent Aleven, Carnegie Mellon University: Supporting Self-Regulated Learningwith Intelligent Tutoring Systems (20)

More from eMadrid network

More from eMadrid network (20)

Recently uploaded

Recently uploaded (20)