Learning objectives: By the end of this workshop, you will be able to: Define versatility, reliability, validity, difficulty, and item discrimination in terms of assessment improvement. Manage built-in Canvas learning management system tools to improve the accuracy of your online exams. Describe some basic strategies for improving the quality of assessment questions.

1. Leveraging Canvas Quiz Analysis to
Improve Your Online Assessments
Justina Brown
Instructional Designer
Center for Instructional Innovation
Academic Technology & User Services
Michael Wilder
Assistant Director
Academic Technology & User Services

2. Active Minds Changing Lives
Learning Objectives
By the end of this workshop, you will be able to:
● Define versatility, reliability, validity, difficulty, and item
discrimination in terms of assessment improvement.
● Manage built-in Canvas learning management system tools to
improve the accuracy of your online exams.
● Describe some basic strategies for improving the quality of
assessment questions.

3. Active Minds Changing Lives
First Things First
What is your role at your institution?
What are you hoping to learn?
What is your experience analyzing quiz questions?
What would you do with an item analysis?
CAVEATS
● Limits of our knowledge
● Limits of item analysis (100 q’s/1000 subs.)

4. Active Minds Changing Lives
What are your experiences with online quizzes?
Instructors:
● What level of cognition (recall,
application, and/or analysis) do
your quizzes require from your
students? Is this level appropriate
for your content?
● Do your assessments accurately
assess students’ mastery?
● How have you used Canvas to help
improve your quizzes?
● What prior knowledge do you have
for improving online assessments?
Instructional Designers/Technologists:
● What are the biggest challenges
in support of online assessment?
● Do you know how to help
instructors improve quiz
questions?
● What is your familiarity with
Canvas quiz tools?
● What types of advice or resources
do you provide?

5. Active Minds Changing Lives
Versatility
Model created by: Rex Heer
Iowa State University
Ctr. for Excellence in Learning and Teaching
Updated January, 2012
Licensed under a Creative Commons Attribution-
NonCommercial-ShareAlike 3.0 Unported License.
For additional resources, see:
www.celt.iastate.edu/teaching/RevisedBlooms1.htm
l
Providing a variety of
different questions
across variety of
cognitive levels of
learning.

6. Active Minds Changing Lives
Reliability & Validity
● Reliability: measure of a test’s consistency
○ within itself (questions yield similar answers)
○ across time (re-tests yield similar results)
● Validity: whether or not the assessment
measures what it claims to measure.
○ Watch out for confusing vocabulary, poor question wording,
or reliance on cultural knowledge.
Ensuring quizzes are valid and reliable prevents potential barriers
to knowing whether students know your content.

7. Active Minds Changing Lives
Difficulty
● Difficulty index: how hard it is to answer the question correctly.
○ This value is typically represented as a counterintuitive
value between 0 and 1.0.
○ Zero is considered hard and 1.0 is easy.
Questions that are either too hard or too easy
provide little information about discrimination.

8. Active Minds Changing Lives
Discrimination
ITEM DISCRIMINATION INDEX
This metric provides a measure of
how well a single question can tell the
difference (or discriminate) between
students who do well on an exam and
those who do not.
The discrimination index indicates the extent to which success
on an item corresponds to success on the whole test.
DISCRIMINATION
VALUE
INTERPRETATION
Above 0.40 Excellent/High
discrimination
0.20 - 0.40 Good/May be improved
0.0 - 0.20 Unacceptable
0 Flawed or keyed incorrectly

9. Active Minds Changing Lives
Discrimination
Students with a better understanding
should do better.

10. Active Minds Changing Lives
Item Analysis in Canvas
This allows you to
winnow your exam down to the items
that best indicate comprehension
of the overall topic.

11. Active Minds Changing Lives

12. Active Minds Changing Lives

13. Active Minds Changing Lives

14. Active Minds Changing Lives
Discrimination & Difficulty in Canvas Quizzes
➢ Example in Canvas

15. Active Minds Changing Lives
Improving Quiz Questions
➢ Ensure questions are clear and not confusing:
○ Clarify the stem.
○ Limit answer choices to 3 or 4.
○ Make answer choices fairly parallel.
○ Make distractors incorrect, but plausible.
○ Limit use of always, never, all of the above, and none of the above.
○ Avoid use of language that has not been defined yet or that requires
a particular cultural background.
➢ Eliminate questions that do not measure intended learning outcomes.
➢ Try different versions of questions to measure learning outcomes and
evaluate those as a group.

16. Active Minds Changing Lives
Improving Quiz Questions - parts
From “Writing Good Multiple Choice Test Questions,” Vanderbilt University

17. Active Minds Changing Lives
Improving Quiz Questions - stem
From “Writing Good Multiple Choice Test Questions,” Vanderbilt University
The stem should be meaningful by itself.

18. Active Minds Changing Lives
Improving Quiz Questions - alternatives
From “Writing Good Multiple Choice Test Questions,” Vanderbilt University
Alternatives should be plausible.

19. Active Minds Changing Lives
Improving Quiz Questions - complexity
From “Writing Good Multiple Choice Test Questions,” Vanderbilt University
Avoid complexity of choices in multiple choice questions.

20. Active Minds Changing Lives
Improving Student Learning
➢ Consider incorporating feedback to help solidify student understanding.
➢ Consider following up with group quizzes.
➢ Consider techniques that encourage critical thinking:
○ Premise – Consequence - Students must identify the correct outcome of a given circumstance.
○ Analogy - Students must map the relationship between two items into a different context.
○ Case Study - A single, well-written paragraph can provide material for several follow-up
questions.
○ Incomplete Scenario - Students must respond to what is missing or needs to be changed
within a provided scenario.
○ Problem/Solution Evaluation - Student are presented a problem and a proposed solution.
They must then evaluate the proposed solution based upon criteria provided.

21. Active Minds Changing Lives
Resources
➢ Canvas Guides
○ Quiz Statistics
○ Canvas Quiz Item Analysis
➢ 10 Examples of Question Improvements
➢ Improving Validity of Large-scale Tests: Universal Design
and Student Performance
➢ Writing Good Multiple Choice Questions
➢ Writing Multiple-Choice Questions that Demand Critical Thinking
➢ WWU Teaching Handbook
○ Crafting Questions
○ Creating Tests
○ Helping Students Learn from Tests

22. Questions & Ideas

23. Active Minds Changing Lives
Feedback
Your feedback is valuable to us!
tinyurl.com/evaluation-wwu
This presentation available here:
https://goo.gl/FDyjtP