4. Sub UniquePostsRead()
For k = 1 To MaxUser Step 1
RowCount = Range("A1").CurrentRegion.Rows.Count
For w = 1 to MaxWeek Step 1
StartTime = Sheets("Week").Cells(w + 1, 2)
EndTime = Sheets("Week").Cells(w + 1, 3)
PostNum = 0
PostsIndex = 0
Do While Cells(i, datestamp) <= EndTime And i <= RowCount
If Cells(i, Source) = “Read" Then
If Cells(i, Message_Author) <> Val(ActiveSheet.Name)
And Cells(i, Scan) <> "X" Then
flag = 0
For j = 1 To PostsIndex Step 1
If Posts(j) = Cells(i, Message_Id) Then
flag = 1
j = PostsIndex
End If
Next j
If flag = 0 Then
PostsIndex = PostsIndex + 1
Posts(PostsIndex) = Cells(i, Message_Id)
End If
End If
End If
Sheets(“Stats").Cells(Line, 22) = PostsIndex
Next w
Next k
End Sub
PercentPostsRead =SUniquePostsRead
TotalPostNumber
13. Learning Analytics
is the development and application
of data science methods to the
distinct characteristics, needs, and
concerns of educational contexts
for the purpose of
better understanding &
supporting learning
14. Learning Analytics
is a critical emerging technology of
the 21st century, with high
expectations of short- and long-
cycle positive impacts on learning
and teaching
25. DEMOGRAPHICS ( W H O P E O P L E A R E )
PERFORMANCE ( H O W T H E Y ’ V E D O N E )
ACTIVITY ( T H I N G S P E O P L E D O )
LO G - F I L E S , P H Y S I C A L T R A C K S , S E L F - R E P O R T
ARTIFACT ( T H I N G S P E O P L E C R E AT E )
P R O B L E M A N S W E R S , W R I T T E N E X P L A N AT I O N S
ASSOCIATION ( C O N N E C T I O N S P E O P L E M A K E )
W H O A N D W H AT T H E Y I N T E R A C T W I T H
DATA
3A’S
WISE (2019)
HOPPE (2015)
27. S I N H A E T A L . ( 2 0 1 4 )
THE KEY IS
TO CONNECT
LOW-LEVEL
BEHAVIORS
WITH
HIGH-LEVEL
CONSTRUCTS
F L I P P E D C L A S S V I D E O V I E W I N G
28. S I N H A E T A L . ( 2 0 1 4 )
THE KEY IS
TO CONNECT
LOW-LEVEL
BEHAVIORS
WITH
HIGH-LEVEL
CONSTRUCTS
Raw
Clicks
Aggregate
Features
Critical
Concept
Info
Process.
Play
SeekFwd
ScrollFwd
RateFast
Skipping Disengaged Low
Play
Pause
SeekBw
SeekFwd
Checkback Searching
for specific
info
Med
Play
Pause
SeekBw
Rewatch Reviewing
content
High
Play
Pause
SeekBw
ScrollBw
Clarify
Idea
Tussling
with
content
Very High
F L I P P E D C L A S S V I D E O V I E W I N G
33. MEANINGFUL VARIABLES TO
CONSTRUCT + INCLUDE
POTENTIAL CONFOUNDS,
SUBGROUPS, OR COVARIATES
WHICH RESULTS TO ATTEND TO,
WHAT THEY MAY MEAN, WHERE
THEY MAY GENERALIZE TO
HOW TO TAKE ACTION BASED
ON OUTCOMES
W I S E & S H A F F E R ( 2 0 1 5 )
THEORY
GIVES
GUIDANCE
34. ONLINE DISCUSSION
LEARNING MODEL
Externalizing one’s
ideas by contributing
posts to an online
discussion
Taking in the
externalizations of
others by accessing
existing posts
• Social constructivist perspective - online discussions as a forum
for learning through conversation
• Students learn as they articulate their ideas, are exposed to the
ideas of others, and negotiate differences in perspective
• Focus on how students contribute comments (“speak”) and
attend to other’s messages (“listen”)
35. Criteria Metric Definition
Listening
Breadth
Percent of posts viewed
Number of unique posts that a student viewed divided
by the total number of posts in the discussion
Percent of posts read
Number of unique posts that a student read divided by
the total number of posts in the discussion
Listening Depth Percent of real reads
Number of times that a student read a post divided by
the total number of times they read and viewed posts
Listening
Reflectivity
Number of reviews of
own / others’ posts
Number of times a student revisited posts that they had
made / viewed previously in the discussion
Conversational
Distribution
Posts made / viewed
throughout discussion
Dispersion or concentration of posts made / viewed by a
student in the discussion space
Speaking
Quantity
Number of posts
Total number of posts a student contributed to the
discussion
Average post length
Total number of words posted by a student divided by
the number of posts they made to the discussion
ONLINE DISCUSSION
LEARNING MODEL ANALYTICS
36. Listening Reflectivity
• Reviewing others’ posts multiple times predicts greater
responsiveness
Listening Depth
• A greater % of real reads predicts richer argumentation
(Informed) Listening Breadth
• Reading a greater % of posts and viewing a greater % of posts
than those read predicts richer argumentation
ONLINE DISCUSSION
LEARNING INSIGHTS
38. Metric Your Data
(Week X)
Class Average
(Week X)
Observations
Range of participation 4 days 5 days
# of sessions 6 13
Average session length 33 min 48 min
% of sessions with posts 67% 49%
# of posts made 8 12
Average post length 386 words 125 words
% of posts read 42% 87%
#of reviews of own posts 22 13
#of reviews of others’ posts 8 112
EXTRACTED LISTENING ANALYTICS
39. CHANGES OVER TIME
“I found that I wanted the challenge of trying to up the % of overall posts
that I reviewed each week. This also meant slowing down my reading since
the data would not record a quick read of the information. The overall result
was that I think I learned more and was able to get a broader sense of
opinion concerning the readings.”
40. Adapted from Galaxy by Image by Gerd Altmann from Pixabay
professional
clinical
becoming
dentistry
Multidimensional
Word Embedding
Space of
Dental Student
Reflective Writing
45. Interpret Data
Sense-Making Pedagogical Response
Get Oriented/
Focused Attention
Find Absolute & Relative
Reference Points
Read Data
Triangulate
Contextualize
Make Attribution
Explain Pattern
AFFECTIVE PROCESSES
Area of
Curiosity
Question
Generation Wait-and-See
Reflect on
Pedagogy
Check
Impact
Take Action
Whole-Class Scaffolding
Targeted Scaffolding
Revise Course Design
Teaching with Analytics
46. Pedagogical
Questions
Are my students
preparing?
Which of my
students need
help?
How are my
students thinking
about STEM?
Data-Based
Answers
Student interaction
grid
Predictive
modelling
Concept network
examination
Educational
Action
Emphasize
important
resources
Offer help in a
targeted manner
Evaluate / update
curricular design
50. Who Are My Students Engaging With?
Always Same People Always Different People
My Actual Class
Avg Degree = 3
Modularity = .81 Avg Degree = 10
Modularity = .14
Avg Degree = 9
Modularity = .27
52. StatMed’13
StatMed’14
StatLearn
YBW
PSY
Course Subject Learning Process Question Words Connectors Existence/Condition Course TasksQuality/Quantity Effort / Action People Appreciation/Greeting
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Content - Related
NonContent - Related
Numberoffeatures Content-Related
Discussion Posts
Non-Content-Related
Discussion Posts
Question Words + Connectors
e.g. “can” “does” “why” “how”
“which “and” “of” “than” “is”
Course Tasks + People
+ Appreciation / Greetings
e.g. “answer” “exam” “course”
“lecture” “thank” “good” “I” “my”
Top Feature Distribution by Category
53. How am I Facilitating Interaction?
U1
• Responses at all levels
• Coaching and supporting
• Social presence cues
“That is correct -
Nice! So how
would you use
this to solve the
question?”
“A bell shape is
not necessary.
You could have
a bimodal
distribution”
U417
• Responses only to top level
• Straight forward answers
• Little social presence
Degree = 4.4
Weight = 2.2
Degree = 2.8
Weight = 1.8
54. SAT Math
Score
Diagnostic
Test
High School
GPA
Race /
Gender Drop, Fail
Withdraw
Grade < C+
Drop, Fail
Withdraw
Grade < C+
Which of My Students Will Need Help?
Introductory Calculus as a Challenging Gateway Course
Beyond identifying who is likely to need help, the model
indicates that the kind of help is support for algebra skills
55. How Are My Students Thinking About…?
D1 D4
Becoming a Dentist
Start D1 End D4Start D3
56. How Are My Students Thinking About…?
D1 D4
Becoming a Dentist
Start D1 End D4Start D3
“I started dental school with the focus of
wanting to become an excellent healthcare
provider for my patients. I did not fully
understand what that meant until D3 and
D4 year, when I had my patients' oral
health needs in my hands.”
58. The Six Elements of Reflection (Wise, Cui & Allen, 2019)
Reflection
Element
Definition
Description Student indicates what they observed about a learning event
Analysis
Student makes sense of the learning event and/or why something
occurred
Feeling Student describes affective reaction to the learning event
Perspective
Student demonstrates change in their own perspective or consideration
of other perspectives
Evaluation Student describes what was good and bad about the learning event
Outcome Student describes lessons learned and future intentions / plans
66. Developing a Model to Automatically Assess Reflective Qualities
The Six Elements of Reflection
Reflection
Element
Linguistic Features (using LIWC)
Model
Accuracy
Description I, we, see, hear, feel, past focus 77%
Analysis
Analytic index, insight, cause, discrepancy,
tentative, certainty, difference
64%
Feeling
I, we, affect, positive emotion, anxiety, anger,
sadness
67%
Perspective She, he, they 60%
Evaluation - N/A
Outcome Analytic index, I, future focus 93%
67. How Are My Students Thinking About…?
Failing an Competency Assessment
Top Words
Overall
68. Analytics in Physical Spaces (by Xavier Ochoa @ NYU)
• Provides an automated
report to students
about their basic oral
presentations skills
• Such as: looking at the
audience, body
posture, speaking
volume and pauses
• Tool for preparing for
class presentations (not
automated assessment)
69. Image Credit: Dakotilla via Flickr (CC BY 2.0)
The clock is a powerful machinery that created
the product of “seconds” and “minutes”
and thus changed our relationship to time
(Mumford, 1934)
70. Image Credit: Dakotilla via Flickr (CC BY 2.0)
How will analytics change our
relationship with our students and
their relationships to (active)
learning process?
How can data inform (w/o dictating)
our pedagogical decisions?
How can our pedagogical decisions
generate better data?
What dangers must we be on the
watch for?
Learning analytics is a powerful machinery that
creates new products of data that change our
relationship to teaching and learning
71. Image Credit: Dakotilla via Flickr (CC BY 2.0)
Ownership, Consent & Choice
Agency & Context
Transparency & Accountability
Privacy & Surveillance
The Right to Be Forgotten
The Creation of New Labels
Learning analytics is a powerful machinery that
creates new products of data that change our
relationship to teaching and learning
72. Learning Analytics at NYU
NYU Learning Analytics is a collaborative effort, focused on
community change that puts people, not data, first
Key Characteristics
• Partnerships between IT, faculty, administrators and students
• User-Centered Design involving stakeholders from the start
• Scalability to serve a large university with 10+ global
campuses and a diverse international student body
• Research to innovate and build a knowledge base for data-
informed teaching and learning in higher education
73. What is NYU-LEARN?
• The Learning Analytics Research Network -> An innovative
new model for synergistic learning analytics research and practice in
the university.
• A supportive and inclusive university wide community
of faculty, instructors, technologists -> United with the
common goal of building the future of data-informed teaching and
learning research at NYU
• A collaborative partner -> For those who want to incorporate
learning analytics into existing research or develop new efforts
74. Predicting
Calculus Success
• Can detect “students
who will struggle”
• Working on effective
intervention strategies
Instrumented
Learning Spaces
• NSF CIRCL subaward to
hold working group
• Generate shareable
multistream data corpus
Many Local Projects
NYU Learning Analytics
Instructor’s Use
of Dashboards
• Documenting practices
of use -> support model
• Fed back to influence
tool design + support
Presentation
Feedback Tool
• Automatic feedback to
student or oral
presentation skills
• Introduce to pedagogy
Dental Reflection
Analytics
• Detecting rise of
patient-centeredness
• In conversation about
impact and action
Interpret Data
Sense-Making Pedagogical Response
Get Oriented/
Focused Attention
Find Absolute & Relative
Reference Points
Read Data
Triangulate
Contextualize
Make Attribution
Explain Pattern
AFFECTIVE PROCESSES
Area of
Curiosity
Question
Generation Wait-and-See
Reflect on
Pedagogy
Check
Impact
Take Action
Whole-Class Scaffolding
Targeted Scaffolding
Revise Course Design
Sense-Making Response
D1 D4
Hjl https://www.flickr.com/photos/hjl/8065776393
Mention some false dichotomies though –
GOOD big data isn’t necessarily easy and incidental – may need to plan systems
To get to causality need to use in combination with some form of experimental design (may be quasi, or over time, but still)
Somewhere pull in that one size doesn’t fit all
Log-files – LMS, classroom response (clickers), hw sets, online textbooks
Physical space (card logs, wired classroom
Great way, when approaching a new situation to think about what might be collected
Levels of specificity – learning model but not soooo specific to the type of discussion – some built in flexibility for transferof situaitons.
High student overall buy-in to guidelines / metrics, was difficult to isolate the two as students seemed to think of them together
Students interpreted metrics in terms of the guidelines
Students described using the guidelines and metrics to decide how to participate
Students found goal-setting valuable as motivating them to improve, used multiple strategies, drew on metrics and tried to adjust behaviors
Validation and surprises - emotional reactions No major “big brother” issues
Involuntary propensity to target average
High student self awareness of if meeting goals
Having an audience for the journal mattered
Negotiation and contextualization of analytics - students explained choices, strategies, struggles
Instructor responses seen as supportive, providing guidance to help students move towards goals
Does this challenge agency? Some tensions…
(“One thing I try to do is actually do it – go into the discussions when I had time to really actually think about things as opposed to just, you know, read them, check, read them, check.”) - purposefulness
Dental - reflection analytics, research phase with 1 conference paper and 1 journal paper accepted, next steps to expand analysis and figure out action impact plan
FAS - calculus project moving forward with a focus on action-ability of the predictive model, also a partner in participatory design of a student-facing learning analytics tool
Writing analytics
Discussion forum analytics - SPS
Presentation analytics -> translation to doctor-patient interactions with med school
Tandon – gradescope cs possibility
Calculus pic: https://pixabay.com/photos/differential-calculus-board-school-2820657/ Free for commercial use. No attribution required.
Other images all our originals.