1. EMERGING PRACTICES IN
E-ASSESSMENT
NAV HUNDAL
LEARNING TECHNOLOGIST
FACULTY OF MEDICINE DENTISTRY & HEALTH
UNIVERSITY OF SHEFFIELD
2. WHAT WILL WE COVER
Who are we?
What drives us to develop our online learning and e-assessment?
What challenges are we addressing?
Key areas of e-assessment focus
• Collaborative e-Assessment
• Portfolio based e-Assessment
• e-Submission, e-Marking & e-Feedback
What are we doing in relation to these themes, feedback and
recommendations
What does the future hold?
3. FACULTY OF MEDICINE
DENTISTRY& HEALTH
Students 2013/14
Undergraduate 2382
(FT 2104 PT 278)
Postgraduate 1192
(PGR FT 216 PT 89)
PGT FT 359 PT 528
Distance Learning
Masters in Dental Public Health
Masters in International Health Technology Assessment
MSc in LACIC (Language & Communication Impairment in Children
MSc Speech Difficulties
Masters in Public Health
MRes Musculoskeletal Ageing
MMid Midwifery
4. DRIVERS & AIMS
Inline with Faculty Strategic Plans such as ‘Learning & Teaching Strategy’ and
‘eLearning Strategy’
- Promotion of learning, teaching and assessment using technology
- Sustainable and flexible structures and online environments
- Increasing PGT numbers
- Increasing flexibility and accessibility to resources
Annual Reflections, feedback from staff and students.
Institutional lines of inquiry into electronic assessment methods and tools.
Widespread use of Turnitin
Further embed Turnitin and Grademark
Portfolio based systems
5. CHALLENGES
• Providing an engaging online
environment for group work and
collaborative working.
• Using technology that is easy to
use and accessible for students
based across different locations
around the globe.
• Providing an e-submission
mechanism for students to submit
assignments online.
• Providing a mechanism to deliver
7. COLLABORATIVE
E-ASSESSMENT
Trends
• Moving away from
VLE tools but still
administering and
instructing via
VLE.
• ‘ a Google
institution’
• Making use of full
app suite
• Collaborative
assessment
Tensions
• Learning new
tools
• Google seen as
third party
• Hangouts
• VLE glitches
8. FEEDBACK &
RECOMMENDATIONS
• Raising awareness of Google for education to staff and
students.
• Stigma of 3rd party
• Blend of tools
• Benefits of ease of use – some students are using Google
already
• Little staff development
• Student choice – creating communities and flexibility
“I feel supported and part of a group”
“Enables me to add more variety to the online programmes and
encourages inclusion and interaction between students”
9. EVIDENCE OR PORTFOLIO
BASED E-ASSESSMENT
Trends
• TNA
• Supervisory
meetings
• Replacing
treatment and
clinic logbooks
• Student
progress
monitoring
Tensions
• Engagement
• Students
needed to see
why they are
doing it, big
task.
• Staff time
management
Dentistry Portfolio
• Bespoke e-portfolio
for UG
• Student centered
• Recording treatments
• Automation of log
books
• Staff and student
monitoring of
progression
• Entries are visible,
recorded and they
get feedback!
10. FEEDBACK &
RECOMMENDATIONS
• Make it meaningful
• Explain why they are using the tool! Portfolios can be
quite a task
• Interact with students, feedback on their evidence
• Keep it student focused
“Advantages include the flexibility to allow students complete this in
conjunction with ongoing research projects. Disadvantages include
getting the students to engage with it initially”
“Students have better knowledge of their own progress”
“More of a flexible way of undertaking tutorial data collection of a physical
event”
11. E-SUBMISSION, E-MARKING &
E-FEEDBACK
Tensions
• Offline marking
• Marking on the
iPad
• On screen
marking
• Double Blind
marking
Trends
• Using through VLE
• Maybe preferred
over from VLE
submission
• Grademark for
richer feedback
• Quickmark
comments
12. TECHNIQUES
1. Second Marking – Feedback provided by first marker in
Grademark. No grade given. Grade held in shared spread
sheet. Second marker grades, discussion between two.
Mark delivered via VLE.
2. Moderation – Feedback via Grademark. Marks and feedback
held in a Google spreadsheet shared between moderators
and markers.
3. Print and go paper based post submission!
External examiner access
13. RECOMMENDATIONS
& FEEDBACK
• Staff development
• Changing marking practice – marking online
• Student anxiety – fear of the originality report!
• Benefits of different modes of feedback within one tool
“Using Grademark has made in-text commenting a lot quicker/easier and while I know some faculty
don't like having to be screen-based for marking (they like the break from a computer screen by
doing paper reading/marking), I actually prefer not having so much paper to deal with - it makes me
feel more organised (whether or not I really am!).”
“Encourages in text comments; initial concerns years ago that it restricted WHERE you could mark,
with several staff saying they used to mark on trains, or in other places where IT access was
limited. Agree but quickly adapt.”
“Students are generally happy that they do not have to travel to the University with two printed
versions of their work. Submitting online saves them travel costs, time and paper.”
14. ON THE HORIZON…
OSCE examinations via iPad.
• Reliability of marking
• Storing marks and scores
• Feedback for students
• Making the scores meaningful
• Software that allows integration with other systems
• Software that allows analysis of scores
• Wifi – offline syncing capabilities
What are we encouraging and supporting?
• Greater use of Tii/Grademark
• Promoting different learning technologies for e-assessment -TEL FEST
• eLearning SIG and Teaching circles for e-Assessment sharing
• Sharing techniques
• Raising awareness of different methods of eFeedback
We are the Faculty of Medicine Dentistry & Health. We have departments such as the ScHARR, Dentistry, Medicine, Nursing, Oncology, Orthotics, Human Communications Sciences, Human Metabolism, Cardiovascular Science and Neuroscience.
Our undergraduate population for 2013/14 – 2382.. … our UG student population is pre set in terms of student number we can take on. Breakdown and our PG. We have a mix of post graduate taught and post graduate research, as you can see see mix of our FT and PT PGT students is heavier on the PT route. And over the years we have seen an increase of part time PGT students coming into the faculty to study whilst working in their professional roles within Medicine, Dentistry and Health
Insert data?
So we have a lot of PT PGT students and one way of attracting our students is by offering a range of Distance Learning courses that they can study whilst working. We have a mix of DL programmes within ScHARR, Medicine, Nursing, Orthoptics, HCS and Dentistry …. Here are just a few well established programmes to give you an idea of the sorts of DL programmes we offer across the departments.
As a driver we as a faculty are keen that that PGT grows, therefor keen to carry on developing our DL provision as a way of attracting working health care professionals.
And this growth and supporting the growth of DL is eflected in many ways within Faculty strategic Plans such as our e-Learning Strategy and Learning and Teaching Strategy. The use of innovative technologies to support LTA is something that is central in delivering these aims. Supporting staff to make these developments is something that is important to us to continue and we do this through of sharing practice, growing learning technologist provision and staff and curriculum development.
Some of the key points in relation to this from theses strategies are to…. (read bullets)
Annual Reflections feedback from Departments highlights current e-assessment specifically as well as feedforward info ----- So why are we interested in e-assessment?
We have various challenges that we face in delivering online learning and e-assessment.
Some of the common challenges we face (and I’m sure many of us face)….. Include the following:… for each one state where we are at now.
We currently use Blackboard as our institutional VLE and are also a ‘Google Uni’
We have the VLE MOLE, Google tools and webinar technologies that aid us in doing this
We use the VLE assignments tools, Tii and some departments are still doing some form of paper submission
And again we have Tii/ Grademark and the assignment tools in MOLE, some staff use email and annotate electronically through MS word etc too.
I’m going to talk about e-assessment in relation to 3 themes which seem to be growing within the Faculty. These are:
Collaborative e-assessment
Portfolio or evidence based e-assessment and reflection
eSubmission, eMarking and efeedback
I’ve picked these key areas of eAssessment to focus in relation to the strand emerging practices, techniques and tensions as each area of e-assessment is developing for us, and this is happening due to technology and tools changing, student feedback and changing practices and in some ways each have their own challenges when it comes to the delivery of e-assessment.
Collaborative e-Assessment
Group work forms a large part of our programme assessment in one form or another, and what we are finding is since the introduction of Google Apps we’re able to use a host of flexible creative apps to deliver this. We have a few strong examples, for example the Health Promotion Module in Dentistry is using Google Docs in stead of the typical VLE wiki tool. Administration and instruction of the task is delivered with via the VLE, so students are allocated groups within the VLE using the randomization of groups ‘tool’, they are given a google Doc to perform a group task and collect and collaborate within the document. Interestingly they also had a discussion forum within the VLE and used this area to go back and discuss their task rather than using the comments facility within a google doc.
Similarly we have just taken this one step further with orthoptics, although this piece of assessment is set to run next month and what we are setting up there is a google site. So students are allocated case studies around ethical issues surrounding the subject, it could be patient care or using social media, and rather than using Google docs to replace the wiki we have created a google site and each group of student will be given a page on the google site. This allows the students to collaborate online, they are able to discuss using the page comments, they are free to choose what ever kind of format of content which they will construct around their case study. Using google sites allows them to use text, link to other resources, docs, presentations and videos. So we are hoping this assessment will produce a really nice end resource. Once the groups have completed the task we will open up the page restrictions and they can view eachothers work. Traditionally students would use a wiki, and need to do a final presentation and an individual essay. They are still required to do this, but they will be given more flexibility to present their site instead, or a video etc.
And we have a couple more examples of using Google docs in Nursing and Scharr in a very similar way. So what our Google Apps suite is allowing us to do is give the students a little more flexibility in the way in which they want to produce a resource for assessment, so allowing for different levels of creativity and learning styles, but also we are finding that students really enjoy using Google as it’s a tool they are already familiar with.
So.. Our trends are….. Our Tensions are
Nursing - Collaboration of Google docs, discussions and connect to explain the assessment.
From following this approach we found: (READ OUT)
Again we have varying types of ePortfiolo based e-Assessment in that faculty, it’s used with our UG, PGT, PGR. We use PebblePad at the university but it’s by no means in full swing. We have had mixed feedback about it and I’ll talk about that in a sec as to why. I’ll just start with our UG Dentistry Portfolio, it’s a bespoke system maintained by a software developer which has designed and developed it. It’s essentially the automation of log books, so the Dentists go off and carry out treatments, they need to have completed a certain amount and different types of treatments, previously all recorded in a paper book, this portfolio has brought all of that online. It’s very much student centered. Staff can then monitor their progress. It takes them time, but they have to be recorded and now everything is visible for staff and students so they get a lot more feedback on their progress.
(clinical log books and targets.... type in their treatments so staff are up to speed with it and monitor
automation of books, student progress...
160,000 treatments
18500 unique patients recorded treatments
undergrad dentistry... hygiene and therapy diploma )
Our main portfolio is PebblePad….
It’s used across PGR and PGT in pockets for things such as TNA, supervisory meetings and progress tracking. For our CIMA MRes in Musculoskeletal Ageing the e-portfolio was used for summative assessment in the TNA process, The aims were to teach students to identify gaps in their knowledge and skills; to identify the training opportunities to fill that gaps; to reflect on the learning experience and plan how to embed what they learnt in their professional life. This was performed throughout the year. We have assessed 5 students. The students had to perform a training needs analysis based on the RDF, prepare a personal development plan with their supervisors, and give evidence of level of skills at the start with examples, review progression in their skills, log any training activity performed and reflection on what they had learnt and what they were going to do nextThe e-portfolio was assessed for the quality of the evidence of the level of skills in the TNA, quality of the PDP and number of activities and for the quality of the reflection.
I think for us this area is something that we will see grow, however right now it seems to be something that takes a lot of thought into administering. And it’s important to use any e-portfolio tool to it’s full potential because they can be a very powerful tool, so not just something as an evidence store. As this seems to be the case in some examples we have found and it’s a time consuming task for students if they aren’t getting anything from it such as feedback on their evidence, reflection and able to see their own progress. This is something we hope to embed further on the CIMA Mres.
The big one. Tii has been around for years and although it different for us in that it’s more established in our faculty as opposed to Google and PebblePad, I’ve decided to talk about it because it’s still a tool that continually is developing, is liked by students and is something we want to embed further in our Faculty. Does everyone here know what Tii is?
Lots of great examples of the use of Tii and Grademark within the Faculty. People like it’s dual functionality, the originality reports and the grading aspect. Grademark provides a rich blend of feedback opportunities for for staff to use, so audio, quick marks, generic feedback and the rubric. We have no policy on whether or not you must use Tii but it seems to becoming standard practice now. AR feedback suggests it’s an area in which students want to have feedback delivered online and via Grademark. It’s accessible, legible, online. And for us, having a robust e-submission mechanism is key for our DL students. So you can see why we use it…. I’m not going to focus to heavily on it’s functionality and benefits because I think it’s something most people know about but I want to look more at what we need to over come in the future to further embed the tool.
Read out Tensions:
Using audio… using ipad for offline marking… lots of staff mark on trains. Still getting people interested in the audio aspect. Many staff think 3 minutes isn’t enough although or scared of the voice recording being stored.
Double blind marking is a big tension as it is for a lot of institutions. Although this is a feature which is in development it has hindered uptake of Tii. We have various work around for this….
How do we get around double blind marking…. Next slide
So a big thing for us and I know for many institutions is the inability to overcome the double marking, or blind marking. These are just techniques, I have no solution until this feature is available in Turnitin! Any other suggestions welcome!
Scharr
Nursing
dentistry
Looking into the future, emerging practices and trends that are developing within the health education sector in-line with the increase of mobile technology adoption. Innovations in mobile application development and the ubiquity of the internet open up a range of possibilities, with this in mind we give consideration to possible future changes in e-assessment for conducting OSCE examinations using mobile devices to overcome some of issues surrounding traditional methods such as reliability of marking, storing scores and providing feedback. In addition to emerging trends we look at the approach taken at both at a faculty and institutional level to support future practice