Top 10 “Must Do's” for Implementing an Assessment Process in Your Program
•Download as PPTX, PDF•
1 like•400 views
Dr. Kyumin Whang, Associate Professor, The University of Texas Health Science Center at San Antonio, School of Dentistry Implementation of a new technology in an educational program is never an easy feat. On-boarding and receiving faculty buy-in can be challenging, but with careful planning is achievable. This presentation will share The University of Texas Health Science Center at San Antonio (UTHSCSA) School of Dentistry’s best practices for implementation, set-up, and construction of assessment procedure to ensure successful adoption and continued use of ExamSoft. Topics covered will include: advantages and challenges of implementation, collaboration with specialists and technicians, setup of support systems, and construction of student academic misconduct procedures.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Top 10 “Must Do's” for Implementing an Assessment Process in Your Program
Similar to Top 10 “Must Do's” for Implementing an Assessment Process in Your Program (20)
More from ExamSoft
More from ExamSoft (20)
Recently uploaded
Recently uploaded (20)
Top 10 “Must Do's” for Implementing an Assessment Process in Your Program
- 1. Top 10 “Must Dos” for Implementing an Assessment Process in Your Program Kyumin Whang, Ph.D., Associate Professor of Biomaterials Department of Comprehensive Dentistry
- 6. Key Results - Institutional student and faculty anxiety Staff support Best practices & exam policy understanding & control of academic misconduct assessment as an educational tool vs. an evaluation tool Analytics for institutional reporting
- 7. Top 10 “Must Dos” for Implementing an Assessment Process 5 10 4 9 3 8 2 7 1 6
- 8. 1 Top 10 “Must Dos” for Implementing an Assessment Process 5 10 4 9 3 8 2 7 Form an ExamSoft Implementation Team 6
- 10. 2 Train the ExamSoft Implementation Team FIRST Top 10 “Must Dos” for Implementing an Assessment Process 5 10 4 9 3 8 7 Form an ExamSoft Implementation Team 6
- 11. 3 Top 10 “Must Dos” for Implementing an Assessment Process 5 10 4 9 Establish School Policy for Assessment Administration 8 Train the ExamSoft Implementation Team FIRST 7 Form an ExamSoft Implementation Team 6
- 12. i. Thou shalt use our new electronic assessment tool for all your exams and quizzes ii. Thou shalt use our new electronic assessment tool for all your preclinical and clinical assessments
- 13. 4 Top 10 “Must Dos” for Implementing an Assessment Process 5 10 Establish School Policy to use Competencies and LOs in Categories 9 Establish School Policy for Assessment Administration 8 Train the ExamSoft Implementation Team FIRST 7 Form an ExamSoft Implementation Team 6
- 18. 5 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy 10 Establish School Policy to use Competencies and LOs in Categories 9 Establish School Policy for Assessment Administration 8 Train the ExamSoft Implementation Team FIRST 7 Form an ExamSoft Implementation Team 6
- 19. Academic Misconduct ✍Randomized Q&A’s help however, students were… ✍ delaying start of exam ✍ copying notes from computer onto worksheet before starting exam ✍ texting answers when “Secure Review” was enabled ✍ copying off other’s screens
- 20. 6 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy 10 Establish School Policy to use Competencies and LOs in Categories 9 Establish School Policy for Assessment Administration 8 Train the ExamSoft Implementation Team FIRST 7 Form an ExamSoft Implementation Team Conduct Faculty Training Workshops
- 21. Advantages of Faculty Implementation Focus on student learning vs. physical implementation Different interests in webinars Different uses of analytics Reporting vs. changes to Q construction Tracking of student progress Improvement to LO’s Improvement to lectures
- 22. Faculty Training Workshop Agenda Session 1: 1) ExamSoft portal 2) Importing and creating LO’s in categories 3) Importing, creating and tagging questions and rubrics 4) Advantages and disadvantages of different question types 5) How to write good LO’s Session 2: 1)Creating assessments 2)Exam Policy 3)Taking a mock test 4)Data analysis 5)Adjusting scores 6)Creating student feedback and reports 7)Computer troubleshooting during an exam
- 23. 7 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy 10 Establish School Policy to use Competencies and LOs in Categories 9 Establish School Policy for Assessment Administration 8 Train the ExamSoft Implementation Team FIRST Motivate Faculty Form an ExamSoft Implementation Team Conduct Faculty Training Workshops
- 24. Motivate Faculty Use of different question typesimproved assessments. Immediate grading Very quick grade adjustments Feedback function Immediate release of scores Release of student performance linked to LO’s instead of questions
- 25. Improved Feedback to Students Competency
- 26. Improved Feedback to Students Subject
- 27. Improved Feedback to Students Instructor
- 28. Improved Feedback to Students Learning Outcome
- 29. Motivate Faculty Use of different question typesimproved assessments. Immediate grading Very quick grade adjustments Feedback function Immediate release of scores Release of student performance linked to LO’s instead of questions LO infofocus on teaching weaker areas LO infohighlight areas students struggle in Review after quizzes based on immediate results
- 30. 8 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy 10 Establish School Policy to use Competencies and LOs in Categories 9 Establish School Policy for Assessment Administration Conduct School-Wide Workshops Train the ExamSoft Implementation Team FIRST Motivate Faculty Form an ExamSoft Implementation Team Conduct Faculty Training Workshops
- 31. Student Training ExamSoft portal Downloading exams Exam procedure How to take an exam Computer troubleshooting Functions in SofTest Uploading exams Checking for grades and how to read reports
- 32. 9 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy 10 Establish School Policy to use Competencies and LOs in Categories Motivate Students Establish School Policy for Assessment Administration Conduct School-Wide Workshops Train the ExamSoft Implementation Team FIRST Motivate Faculty Form an ExamSoft Implementation Team Conduct Faculty Training Workshops
- 33. 10 Top 10 “Must Dos” for Implementing an Assessment Process Create Best Practices and Exam Policy Conduct Regular Computer Maintenance Sessions & Orientation Workshops Establish School Policy to use Competencies and LOs in Categories Motivate Students Establish School Policy for Assessment Administration Conduct School-Wide Workshops Train the ExamSoft Implementation Team FIRST Motivate Faculty Form an ExamSoft Implementation Team Conduct Faculty Training Workshops
- 34. Questions? Kyumin Whang, Ph.D., Associate Professor of Biomaterials Department of Comprehensive Dentistry whang@uthscsa.edu
- 35. Innovations in Assessment • Learn innovative processes in assessment • Network with colleagues • Pre-conference workshops with ExamSoft trainers • Visit www.assessmentconference.com for more details
Editor's Notes
- This is our old Dental School
- We are currently moving to our new clinical facility, the Center for Oral Health & Research, which houses the latest tech ~400 DS students, a dental hygiene program, and various dental residency and graduate programs Ranked #1 DS when last ranked One of first five DS w electronic curriculum/laptop support So we are very innovative and progressive… Conflict: Innovation/Progressive vs. Fear of Technology. The average age of faculty here is 55!
- In 2013, we were still using Scantron. However, ailing Scantron equipment, the high cost of replacement parts, and the closure of the Scantron support facility prompted our school to explore electronic testing. * Initially, Blackboard with Respondus Lockdown Browser was our logical choice since Bb was our LMS. However, we had already had experience with this and found it to be lacking for our application because only ~70 students/time could use it and it was not secure enough. We actually had a student who hacked the system to capture an image of the exam at given intervals while taking a test. So, we looked elsewhere and found that our Nursing & Medical Schools were using Examsoft, so the administration decided to move over to * this with a 1-yr transition. And we started with some 2 examsoft webinars and help from some of the faculty and staff from the School of Nursing and the Office of Blended Learning.
- I got my start with ExamSoft because, being one of the very few Engineers in the HSC, I was the first to try using it in my class with 7 Quizzes (Extra Credit) and 2 Exams and I made all the mistakes first. As such, I was asked to join an Ad Hoc SofTest Implementation Committee, and that led to me leading our first training session at our Faculty Development Day. Then I was asked to help faculty and staff during this transition period. However, as time went by, it was felt that an official ExamSoft “go to” person was needed and I had the privilege of having this second birth at my institution.
- Over the last two and a half years, almost all of our didactic classes use it and there has been a… So, today, I would like to share, from my experiences, 10 “Must Dos” in implementing an assessment process in your institution.
- The first thing that should be done is to form an ExamSoft Implementation Team. This dream team should consist of…
- …preferably, a point person who is a faculty member who is tech savvy. This needs to be an official position and not a volunteer position. Only then will people comfortable asking for help. The rest of the team should include… *people from your educational resources department, making sure that *an assessment expert is included, and someone from *laptop or IT support. Each of these people play a critical role in the success of your implementation
- The next thing is to have this team trained properly FIRST. I stumbled into this role by making a lot of mistakes because I was not fully trained before I used it. We started with two webinars and our initial help came from the SON and OBL but since they were not in our school or on any implementation team initially, and since they were not faculty, people did not seek them out as much as needed. Also, it is difficult to motivate faculty to start on this immediately through two webinars. Training on new software is just not very easy via webinars, especially if the faculty is older. We can see what it can do but we actually need much more time to try it over and over again. So my recommendation is that an implementation team be chosen first and trained BEFORE the rest of the school is even notified of this transition. A resident “expert” faculty and team is a must.
- The 3rd thing we need to do is to establish a school policy for assessment administration. This is before the transition is announced. In my experience, there is so much resistance to change that the decision to switch needs to be top-down. I know a lot of faculty moan and groan about this, but I am a faculty member and, while I understand the legitimate reasons for some moaning and groaning, I believe that if this is actually going to happen successfully, it needs to be a top-down decision, no if, ands, or buts about it. This is what happened at my school and there were many moans and groans but once the transition period was over, everyone had found a way to deal with it. So I am thinking about something like this…
- Just kidding. But if I lived in an ideal world, I would even add this… And get that monster started with. More on this later.
- Along with the official policy to do this, there also needs to be an official policy that requires *ALL teaching faculty to add their learning objectives into the Categories of ExamSoft. The school needs to do its part and add the competencies there too BEFORE anything is announced or anyone other than the implementation team is trained. The second part of the policy then needs to require course directors to link each and every question to the LO’s and competencies. I cannot emphasize this enough! Doing this retroactively is possible but HIGHLY IMPROBABLE! The amount of work needed to retroactively tag each question is huge and you don’t want to do this. You will have a LOT of trouble getting faculty to do this. Now, in my school, all course directors are REQUIRED to have LO’s in their syllabus and/or lecture notes. So, this should not be much extra work. In addition, by linking them to each question, it will force the course director to make sure that what they teach is what they assess and that each question serves to accomplish the LO’s. It actually improves teaching! And on the back end, it will allow us to assess student progress as well as give useful student feedback after an assessment. I will discuss this later.
- So this is how we set up our categories. Initially we only had * these three categories.
- This is what our competencies look like
- Since then I have added my course and subjects to tag my questions with.
- And I added my LO’s nested into each lecture/subject and I tag my questions to that. I have since trained some faculty who want to do this and have given them access to my subjects so they can cross tag their questions w my categories.
- The ExamSoft Implementation Team should then create the best practices and exam policy document. I have one that I can share and I’m sure others have them as well. ExamSoft also has a document for best practices so you can reference those and not reinvent the wheel. In this document, the procedure in administering the assessment is needed as well as tips and pointers to the faculty and student in this process is needed. Finally, tips to prevent academic misconduct is needed since it is a different method of doing an assessment so there are different ways to cheat.
- For example… … so try to make the lengths of the answers about the same so it is more difficult just to see which one the student in front clicked. Also, I now use larger lecture halls for exams or rooms with circular tables. Moving on…
- Only after all this, should the announcement be made and the *faculty trained. One suggestion is to do this at the end of the academic year or during the summer so faculty can get up to speed over the summer and use it in the fall. Regardless, it is important that the implementation team members are experts at this by now. Nothing is more discouraging than having people up front who don’t really know what they are doing or what needs to be done try to teach this. If the team is well-trained, it will instill confidence in the faculty that they will be well-supported. Now, I want to emphasize that the faculty needs to be trained before others because what I saw was that when staff was involved early, many of the faculty did not bother to learn how to use this and proceeded to dump the work on them. The problem with this is that there are many advantages to having faculty be heavily involved in the use of examsoft rather than the staff. Sure staff can help but the faculty really needs to know how to use it and what it is capable of. I have a white paper on the ExamSoft site that explains this in more detail, but briefly….
- … We need the course director to do this to take full advantage of the data we can get from this.
- For these workshops, I suggest breaking it up into at least two separate sessions. The first session should include the following: an introduction to the ExamSoft portal How to import and create LO’s in categories how to import and create and tag questions and rubrics Advantages and disadvantages of different question types *If there is more time, this would be an excellent time to train the faculty on how to make good LO’s Then the faculty should be given some time to make and import some questions and then about a week to do that with most, if not all of their questions. The 2nd session should then include: Creating an assessment Exam Policy Taking a mock test Analysis of data Adjusting scores Creating student feedback and reports Computer troubleshooting during an exam
- 7) to motivate faculty. The first motivation is reassuring them that they will have great support from their PP and implementation team The 2nd set of motivations have to do with how it will improve our teaching:
- The 2nd set of motivations have to do with how it will improve our teaching: The use of different questions leading to improved assessments. This includes understanding some test-taking skills Immediate grading Very quick adjustments to grades Feedback function for Q’s and challenges that can be addressed without students coming to talk to you Immediate release of scores increases student visits to faculty Release of category information that are tagged to LO’s help students see what they did not understand without releasing questions
- Category info also helps faculty see what they may need to focus more on when teaching I also use that to highlight where students have trouble in exams and tell the next class when I teach the same lecture again When used as quizzes, we can go over the Q immediately based on immediate results Finally, if at all possible, financial incentives would be helpful in motivating faculty to use different tools to enhance their teaching, like a teaching award that faculty can apply to. This is important because most faculty may end up just using MC’s and not expanding their method of assessments. An example is the use of rubrics for novel purposes such as pre-clinical and clinical performance is still at an infant stage. The way clinical assessment is done, it is not easy to translate to a rubric and much effort is still needed here! All of this can be scholarly activities and ExamSoft provides the data for us to use.
- After the faculty is trained, then staff and students can be trained. Staff training should follow the faculty training agenda. The students only need one session to orient them to the following: Their ExamSoft portal Downloading exams Exam procedure How to take an exam Computer troubleshooting All the available functions in SofTest Uploading exams Checking for grades and how to read reports
- Their ExamSoft portal Downloading exams Exam procedure How to take an exam Computer troubleshooting All the available functions in SofTest Uploading exams Checking for grades and how to read reports
- Now students, while they really don’t have a choice, should also be motivated to use this. This includes: Immediate grade feedback A more detailed report of where they are week, especially if the faculty does not release their exams Increased security from others cheating on them No need to worry about handwriting Ability to send feedback/challenges to the prof w/o worrying about forgetting
- Finally, regular computer maintenance sessions (at least bi-monthly) need to be done so problems w computers can be caught early, and if students do have problems, they can address them at that time. Also, annual orientation sessions need to be done to remind faculty and students of how to use the system. All new faculty should be required to undergo ExamSoft training if they are going to teach at all. I believe that when all of these 10 Must-Dos are in place and continually improved, will have a smooth implementation of a new assessment process.