How to apply data mining techniques to educational research

1. Data Mining in Education
Social Media + Text
Qiang Hao
neohao@uga.edu
http://tobeneo.com

2. Goals
• What is Data Mining?
• What tools / knowledge do you need to do
Data Mining?
• What is the basic process of Data Mining?

3. Questions Answered by Data Mining
• Can we predict whether the coming email is
a spam?

4. Questions Answered by Data Mining
• Can we predict whether the coming email is
a spam?

5. Questions Answered by Data Mining
• Can we predict whether the coming email is
a spam?
money
you
he
……

6. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?

7. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?
#Trump
#DonaldTrump
#GOPTrump

8. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?

9. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?
a, an, the, is, are,
was, were, if …

10. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?

11. Questions Answered by Data Mining
• What is the attitude of people on Twitter
towards the presidential candidate Donald
Trump?
Negative
Neutral
Positive

12. Educational Questions to Answer by
Data Mining

13. Educational Questions to Answer by
Data Mining
• What algorithm can score essays as teachers
do?

14. Educational Questions to Answer by
Data Mining
• What courses should we recommend to
students based on their online activities?

15. Educational Questions to Answer by
Data Mining
• Does the intervention improve students’
lexical variety in their writing?

16. Educational Questions to Answer by
Data Mining
• Are there different patterns in students’
questions; if so, are the patterns related to
their academic performance?

17. Educational Questions to Answer by
Data Mining
• What sub-topics do students tend to cover
when discussing this topic?

18. Educational Questions to Answer by
Data Mining
• What predictor is the most important one for
whether college students seek help online in
their learning?

19. Goals
• What is Data Mining?
Replicable
Reproducible
Automatic

20. Goals
• What is Data Mining?
• What tools / knowledge do you need to do
Data Mining?

21. Tools / Knowledge

22. Tools / Knowledge
Carmen Reinhart Kenneth Rogoff
Thomas Herndon

23. Goals
• What tools / knowledge do you need to do
Data Mining?
Expert level of knowledge in statistics
Intermediate level of knowledge in
programming
Familiarity with R/Python

24. R for SAS and SPSS Users
Robert A. Muenchen
Goals

25. Hands-On Programming
with R
Garrett Grolemund
Goals

26. Goals
• What is Data Mining?
• What tools / knowledge do you need to do
Data Mining?
• What is the basic process of Data Mining?

27. Data Collection
Data Cleaning
Data Processing
Data Analysis
Sharing Data and Results
Research Pipeline

28. Data Collection

29. • XML
Data Collection

30. Data Collection
• JSON

31. Mining the Social Web 2nd
Edition
Matthew A. Russell
Python
Data Collection

32. Data Cleaning

33. Data Processing

34. Data Processing

35. Data Processing

36. Data Processing
Text Analysis with R for
Students of Literature
Matthew L. Jockers

37. Data Analysis
• Lexical Variety
• Classification
• Clustering Analysis
• Latent Semantic Analysis
• Support Vector Machine
• Sentimental Analysis
• Topic Modeling

38. Data Analysis
Renkl, A. (1997). Learning from worked‐out examples: A
study on individual differences. Cognitive science, 21(1), 1-29.

39. Data Analysis
An Introduction to
Statistical Learning
Gareth James
Daniela Witten
Trevor Hastie
Robert Tibshirani

40. Sharing Data and Results
• R + KnitR + RPub
• GitHub

41. Sharing Data and Results
• R + KnitR + RPub:
http://rpubs.com/neohao/online-help-
seeking

42. Sharing Data and Results
• GitHub: https://github.com/Neo-
Hao/TwitterHashtagR

43. Sharing Data and Results
Version control with Git
Jon Loeliger

45. Thanks!