Top Concept Networks in Student Reflections
•Download as PPTX, PDF•
1 like•141 views
Analysis of student reflections using top concept networks.
Recommended
Recommended
More Related Content
Similar to Top Concept Networks in Student Reflections
Similar to Top Concept Networks in Student Reflections (20)
More from alywise
More from alywise (20)
Recently uploaded
Recently uploaded (20)
Top Concept Networks in Student Reflections
- 1. Top Concept Networks of Health Education Reflections Alyssa Wise & Yi Cui Sameen Reza LAK ‘19, Tempe, Arizona March 7th 2019
- 2. • Strategize effective learning to meet goals • Develop path to lifelong learning as a reflective practitioner • Build a sense of professional identity Picture Credits: Susan Merrel 2016 dentistry.ucsf.edu/programs/post-grad/aegd INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Role of Reflection in Health Education
- 3. 2 One Graduated Class (“201X”) Students 367 Practice groups (25-30 stu per group) 14 Reflection Types (each 2x a year) 6 Years of Dental School 4 Potential reflections available for analysis ~17,500 deidentified statements INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Online Dental Portfolio System
- 4. What they contained Focus on “depth” of reflection INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION From Coming-to-Know to Coming-to-Be What they indicated Focus on what reflections tell us about students’ thinking Prior Studies This Study Manual analysis Human coding by experts in the domain Automated analysis Leverage computational approaches Can we identify shifts in students’ conception of the profession and their identity as professionals ?
- 5. Use reflections to track changes in students’ conception of being a dental professional from start to end of program Identify potential linguistic markers to detect changing student conceptions INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Objectives
- 6. Initial Study • Close analysis of 2 Practice Groups • Examined 2 Points in Time • Start of Dental School - First reflection in D1 • End of Dental School - Last reflection in D4 • Qualitative examination to support interpretation Recent Extensions • Repeated study with all 14 practice groups • Examined 8 Points in Time (Start /End all 4 years) • Looked at students individually INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Overview of the Process 54 students 497 reflections 367 students 7510 reflections
- 7. Extract Top Concepts Pre Processing (tokenization, stop words) Create Network of Co-occurrence INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Select Reflections Building Top Concept Networks
- 8. 7 GROUP A Reflections 105 Sentences 475 Average Length 4.5 Unique Words (Nodes) 1,139 Edges 18,618 Average Degree 16 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start D1 End D4 GROUP B Reflections 82 Sentences 283 Average Length 3.5 Unique Words (Nodes) 833 Edges 14,227 Average Degree 17 GROUP A Reflections 154 Sentences 995 Average Length 6.5 Unique Words (Nodes) 1,604 Edges 35,147 Average Degree 22 GROUP B Reflections 156 Sentences 1,176 Average Length 7.5 Unique Words (Nodes) 1,746 Edges 41,375 Average Degree 24 Initial Network Overview
- 9. GROUP A Word Freq % professional (146) 2.80% will (112) 2.15% patient (104) 1.99% become (85) 1.63% dental (76) 1.46% dentist (76) 1.46% want (72) 1.38% learn (60) 1.15% work (59) 1.13% good (53) 1.02% year (40) 0.77% hope (34) 0.65% skill (27) 0.52% 8 GROUP B Word Freq % will (102) 2.99% professional (83) 2.43% patient (76) 2.23% become (57) 1.67% good (52) 1.52% dental (51) 1.50% hope (47) 1.38% skill (47) 1.38% year (46) 1.35% dentist (45) 1.32% learn (34) 0.94% want (17) 0.50% work (12) 0.35% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Year 1 Top Concepts
- 10. GROUP A Word Freq % professional (146) 2.80% will (112) 2.15% patient (104) 1.99% become (85) 1.63% dental (76) 1.46% dentist (76) 1.46% want (72) 1.38% learn (60) 1.15% work (59) 1.13% good (53) 1.02% year (40) 0.77% hope (34) 0.65% skill (27) 0.52% 9 GROUP B Word Freq % will (102) 2.99% professional (83) 2.43% patient (76) 2.23% become (57) 1.67% good (52) 1.52% dental (51) 1.50% hope (47) 1.38% skill (47) 1.38% year (46) 1.35% dentist (45) 1.32% learn (34) 0.94% want (17) 0.50% work (12) 0.35% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Year 1 Top Concepts “I want to use the hard knowledge and skills I learn to further my career” “I know that professionals need to develop necessary inter-personal skills”
- 11. GROUP A Word Freq % patient (349) 3.12% dental (244) 2.18% professional (242) 2.16% good (157) 1.40% learn (147) 1.31% will (146) 1.30% become (123) 1.10% care (114) 1.02% school (107) 0.96% year (103) 0.92% feel (59) 0.53% 10 GROUP B Word Freq % patient (426) 3.17% professional (285) 2.12% dental (221) 1.65% will (199) 1.48% learn (168) 1.25% feel (148) 1.10% care (147) 1.10% good (146) 1.09% year (141) 1.05% become (130) 0.92% school (93) 0.69% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Year 4 Top Concepts
- 12. GROUP A Word Freq % patient (349) 3.12% dental (244) 2.18% professional (242) 2.16% good (157) 1.40% learn (147) 1.31% will (146) 1.30% become (123) 1.10% care (114) 1.02% school (107) 0.96% year (103) 0.92% feel (59) 0.53% 11 GROUP B Word Freq % patient (426) 3.17% professional (285) 2.12% dental (221) 1.65% will (199) 1.48% learn (168) 1.25% feel (148) 1.10% care (147) 1.10% good (146) 1.09% year (141) 1.05% become (130) 0.92% school (93) 0.69% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Year 4 Top Concepts
- 13. GROUP A Word Freq % professional (146) 2.80% will (112) 2.15% patient (104) 1.99% 12 GROUP B Word Freq % will (102) 2.99% professional (83) 2.43% patient (76) 2.23% GROUP A Word Freq % patient (349) 3.12% dental (244) 2.18% professional (242) 2.16% GROUP B Word Freq % patient (426) 3.17% professional (285) 2.12% dental (221) 1.65% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Changes Over Time Start D1 End D4
- 14. GROUP A Word Freq % professional (146) 2.80% will (112) 2.15% patient (104) 1.99% 13 GROUP B Word Freq % will (102) 2.99% professional (83) 2.43% patient (76) 2.23% GROUP A Word Freq % patient (349) 3.12% dental (244) 2.18% professional (242) 2.16% GROUP B Word Freq % patient (426) 3.17% professional (285) 2.12% dental (221) 1.65% INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Changes Over Time Start D1 End D4
- 15. 14 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Concept Connections Group A Start D1 End D4 Group B
- 16. GROUP B Word Freq (%) clinician 20 (0.15%) present 15 (0.11%) issue 14 (0.10%) teach 14 (0.10%) remain 13 (0.10%) attend 12 (0.09%) speak 12 (0.09%) surgery 12 (0.09%) though 12 (0.09%) tooth 12 (0.09%) 15 GROUP A Word Freq (%) graduation 22 (0.20%) encounter 16 (0.14%) clinician 11 (0.11%) comprehensive 10 (0.09%) almost 9 (0.08%) honesty 9 (0.08%) complex 8 (0.07%) fill 7 (0.06%) omfs 7 (0.06%) sometimes 7 (0.06%) INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION “I have grown in knowledge and in skill; I have learned to think like-a-clinician.” Words Used Only in Year 4
- 17. • Overall increase D1->D4 in reflection amount, concepts and connections • Shift from global idea of becoming a good dental professional in D1 to a more integrated patient- centered conceptualization in D4 • Increasing frequency and centrality of patient may be a useful indicator for this (desired) process. • Initial signs in D4 of students starting to think of themselves as members of the dental profession INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Summary of Insights from the Initial Study
- 18. 17 Objectives • Identify if the shift towards patient centered thinking occurs across all students • Identify point(s) in time when shift occurs • Identify students for whom the shift does not occur INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Recent Extensions Data All 367 students in 201X class Across all 4 Years 7510 reflection statements
- 19. 18 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start of D1 End of D4 Top Concept Networks for All Students
- 20. 19 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Top Concept Networks Across 8 points in time D1 Start D1 End D2 Start D2 End D3 Start D3 End D4 Start D4 End
- 21. 20 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Top Concept Network: Start of D1 Start of D1 1020 Reflection Statements
- 22. End of D1 21 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION End of D1 513 Reflection Statements Top Concept Network: End of D1
- 23. 22 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start of D2 95 Reflection Statements Top Concept Network: Start of D2
- 24. 23 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION End of D2 486 Reflection Statements Top Concept Network: End of D2
- 25. 24 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start of D3 940 Reflection Statements Top Concept Network: Start of D3
- 26. 25 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION End of D3 1324 Reflection Statements Top Concept Network: End of D3
- 27. 26 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start of D4 1103 Reflection Statements Top Concept Network: Start of D4
- 28. 27 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION End of D4 2109 Reflection Statements Top Concept Network: End of D4
- 29. 28 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Start of D1 End of D4 Start of D3 Summary of Changes Over Time “I started dental school with the focus of wanting to become an excellent healthcare provider for my patients. I did not fully understand what that meant until D3 and D4 year, when I had my patients' oral health needs in my hands.”
- 30. START OF D1 Word Freq % will 1071 2.62 professional 943 2.31 patient 866 2.12 good 682 1.67 become 681 1.67 dental 649 1.59 dentist 552 1.35 want 509 1.25 skill 474 1.16 hope 455 1.11 29 START OF D3 Word Freq % patient 1321 3.07 professional 804 1.87 good 741 1.72 learn 618 1.44 will 603 1.4 become 531 1.24 dental 504 1.17 year 410 0.95 work 398 0.93 clinic 392 0.91 END OF D4 Word Freq % patient 3936 3.01 professional 2454 1.87 good 2022 1.54 dental 1877 1.43 learn 1822 1.39 will 1814 1.39 become 1403 1.07 year 1348 1.03 care 1242 0.95 feel 1138 0.87 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Degree of “patient” 1.15 Degree of “patient” 2.67 Degree of “patient” 3.53 Identifying Students Who Don’t Undergo Shift
- 31. Student X at Start of D1 30 INTRODUCTION MODEL INITIAL STUDY RECENT CONCLUSION “Interactions and communication with colleagues and patients is also a key. This helps to learn how to become an understanding professional who knows how to achieve their patients needs and communicate in an effective manner.” Identifying Students Who Don’t Undergo Shift
- 32. 31 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION I strongly believe in strong determination, motivation and hard work, which lead me to be a part in dental school. After almost 2 years in clinic I feel that I have improved a lot in my skills and learned various techniques. Student X at Start of D3 Identifying Students Who Don’t Undergo Shift
- 33. • Initial exploration has identified common trajectories of students conception of becoming a dental professional • Recent extensions have provided support to initial results across a larger set of students and pinpoint when the transition happens. • In the process of refining markers that can identify individual students who conform to or diverge from these common patterns. • Refining the notion of concept beyond simple word usage (experimenting with conditional word embeddings) 32 INTRODUCTION MODEL INITIAL STUDY RECENT EXTENSIONS CONCLUSION Conclusions and Further Work
- 34. Thank You Alyssa Wise | Yi Cui | Sameen Reza @NYU_LEARN @alywise alyssa.wise@nyu.edu yca231@sfu.edu sr4918@nyu.edu New York University Learning Analytics Research Network
Editor's Notes
- Reflective practitioner in an ever changing field
- Four boxes
- Narrow focus allowed for careful interpretation of the analytics including follow up of how top concepts were used in the underlying data System level data did not show much difference between groups (same number of reflections submitted)
- Last two steps that we tried a bunch of different things How to determine “top” and define “co-occurance”
- Excited by our initial results were were eager to learn more. The purpose of this extension was 1. VALIDATION. Do the results of the initial study extend across the complete set of students 2. Narowwing when in this 4 year time period does the change begin to appear? 3for the sake of actionable insight we wanted to identify ways to determine indivifually which students conformed to this general pattern of change and which did not Thanks to our Dental portfolio system we had a rich data set that allowed us the affordance to carry out this analysis
- In response to our validation question we looked athe top concept network created at the very start of the program and compared it to that produced at the very end. As you can see the shape of the network as well as the patient node is changes radically from D1 to D4. Thus we felt confident in assuming that the change was indeed taking place across the complete set of students
- In response to our validation question we looked athe top concept network created at the very start of the program and compared it to that produced at the very end. As you can see the shape of the network as well as the patient node is changes radically from D1 to D4. Thus we felt confident in assuming that the change was indeed taking place across the complete set of students
- 2. Students were required to enter reflection once in each academic term. In orderIn order To Identify WHEN it appears to start happening, we collected statements posted at the very first and at the end of each dental year giving us 8 data points in time. lets see these results in a linear fashion from start to end. There were approximately 1000 statements submitted at this data point. The word patient was used quite frequently along with professional will and good
- So here we had only about 500 statements which shows the sparseness of the graph as well. Accordingly patient and professional were both used with nearly the same frequency
- The number of statements collected at this point in time were very low- only about a 100. patient is still connected to good will and dentist but its direct link with professional does not appear here
- The number of statements increase to 500 again Patient is now also associates with learn
- Start of D3.. The number of statements increases to approximately 1000. tHere the graph assumes a radically circular structure and the node patient assumes the central position.
- End of D4… there is not much change here
- Start of D4. Frequent use of “patient” continues to rise
- End of D4. Maximum use of word patient used along with good care learn treat
- Becoming patient centered was an important find. It is something that is listed as a core competency in the dental school objectives. These results showed us that start of d3 isa critical time when this patient centered thinking starts to appear. Knowing that dental students start their clinics in D3 we expected this shift to happen sometime in D3 but what was surprising was that it appeared at the start of D3 rather than later. In students own words…. I…
- We then looked closely at these datapoints in time to explore different ways in which we can identify patterns emerging from students thinking. One interesting feature was that only the use of the word “patient” was increasing from D1 to D3 and remained relatively stable towards D4. Having identified that start of program, start of D3 and End of D4 were the most important times to look at , we took a deep dive into the results from the analysis at these stages. We looked at the common derived results from all students We compiled a list of words that was being used most frequently at each of the three points. Most of the words appeared to be the same. We looked at what different was being said and discovered that students in D4 were mainly using more terms related to their profession and experience of being a dentist. They were talking more about ethical issues, of upholding the prestige of their profession. We tried to identify linguistic markers to see how reflections can help us cluster students who follow this positive trajectory and those that don’t. One thing that came up from our analysis was that very few words were going up in percentage. They were going up in numbers perhaps because of the larger number of reflections but not in percentage. “Patient” and “good” were the only words that went up in percentage from D1 to D3 and while “good decreased in percentage from D3 to D4 patient remained more stable.
- The use fo the word patient was thenemployed to determine students who conformed to the pattern and who did not. Here is an example fof the top concept network of an individual student who was more patient centered in d1 thinking about achieving his patients needs. Switch graph to left With this one feature in hand, we tried to see for which student does this phenomenon not happen. Which students don’t talk more about patients in D3 compared to D1 and this is an example of such a case. This student talks more about patients in D1 than D3 contrary to the pattern that we had observed. The graph is now this individual students top concept graph in D1
- However, s time progresses some students get more involved in their own academic progression. Their focus on patients in less obvious in their reflections. The same student in D3 was not thinking more about his improvement and skill development.
- We are happy that this extension provided support to initial results and help pinpoint a key time in the academic progression where students undergo a change in thinking. We are currently working on refining markers beyond use of words and progress towards concepts. For this we are experimenting how conditional word embeddings can be made useful
- Thank you for listening to our presentation. If you want to get any research update from us or NYU LEARN, please keep contact with us. If you have any question, we’re very happy to answer it. Do you have any question?