4 March 2010 (Thursday) | 15:30 - 17:40 | http://citers2010.cite.hku.hk/abstract/20 | Dr. Barbara MEANS | Center for Technology in Learning, SRI International

1. What Should We Assess with
Technology?
Barbara Means
Center for Technology in Learning
SRI International
A Workshop for the University of Hong Kong
4 March 2010
© 2008 SRI International

2. We don’t measure the qualities on which
our economic competitiveness depends
I’m calling on our nation’s governors and state
education chiefs to develop standards and
assessments that don’t simply measure whether
students can fill in a bubble on a test, but whether
they possess 21st century skills like problem-solving
and critical thinking and entrepreneurship and
creativity.
-- President Barack Obama, March 10, 2009
2

3. ICT and its increasing availability in schools
open up new options for assessment
 More complex items/tasks
 Different kinds of measures
- Interactions with simulations, visualizations,
tools, data sets, and agents
 Embedding assessment within instruction
3

4. Some Terminology
4

5. Some Terminology
Assessment Test
5

6. Test
6

7. Assessment
7

8. Some Terminology
Assessment Test
Formative Assessment
Summative Assessment
8

9. Some Terminology
Assessment Test
Formative Assessment
Summative Assessment
Reliability Validity
9

10. Some Terminology
Assessment Test
Formative Assessment
Summative Assessment
Reliability Validity
High-Stakes Assessment
10

11. Some Terminology
Assessment Test
Formative Assessment
Summative Assessment
Reliability Validity
High-Stakes Assessment
“Drop-in-from-the-Sky” Assessment
11

12. Assessments are important because they . . .
 Express priorities for curriculum and
instruction
 Provide a model of pedagogy that teachers
emulate
 Stimulate curriculum developers to revise
instructional materials to match high-visibility
tests
Source: Binkley et al., January 2010, ATCS21 draft white paper
12

13. BUT high-stakes assessments can lead to . . .
 Schools and teachers focusing on what is tested
rather than the full underlying standards or
learning goals
 A one-time performance orientation and
transmission-style teaching to the test
 Devoting considerable instructional time to test
preparation
Source: Binkley et al., January 2010, ATCS21 draft white paper
13

14. Important aspects of expertise are hard to
capture on multiple-choice tests
 Inquiry and problem solving
 Planning and self monitoring within
complex tasks
 Collaboration
14

15. 15

16. Lake Simulation Shell
 Scenario
– A pristine lake in Costa Rica is slated for development and
will have a new visitor center. Students are to study basic
ecology of lake and make recommendations.
 Driving Question
– What is the basic ecology of the lake?
– How can we use this knowledge
• to educate local citizens and international visitors,
• to inform the development of regulations aimed at sustaining the
lake‟s ecosystems.
 Assessment Task Types
– Observe the organisms, identify their roles and
interrelationships, draw food web
– Conduct simulation-based investigations of impacts of varying
numbers of organisms
Source: Quellmalz & Pellegrino, 2009
16

17. Life Science Simulation
Students
– view animation to observe relationships among organisms
– draw food web illustrating those relationships.
17

18. Life Science Simulation
18

19. Life Science Simulation
In the experiment that you just analyzed, the amount of alewife was set to 20 at the beginning.
Another student hypothesized that the result might be very different if she started with a larger
or smaller amount of alewife at the beginning.
Run three experiments to test that hypothesis. At the end of each experiment record your
data by taking pictures of the resulting graphs.
After three runs, you will be shown your results and asked if it makes any difference if the
beginning amount of alewife is larger or smaller than 20.
19

20. Assessment in the River City MUVE
Enlargement of Microscope
20

21. River City
 Teaches concepts from biology, ecology and
epidemiology
 Students work in teams of three, moving through the
city to run tests in response to the mayors‟ challenge
 Teams keep online journals, analyze data, form
hypotheses, and write up their research in a report for
the mayor
 Researchers developed measures of science concept
knowledge, science inquiry skills, and sense of efficacy
as a scientist based on student actions within the River
City environment
21

22. River City’s Log File Captures . . .
 Where a student went
 With whom the student communicated and what was
said
 The artifacts the student activated
 Databases the student used
 Data that the student gathered using virtual scientific
instruments
 Screenshots and notations the student entered in a
virtual notebook
 Hints that the learner requested
Student learning measures were created from these data.
22

23. Pilot Interactive Learning Assessments from
the LIFE Center
Source: Svihla, Vye, Brown, Phillips, Gawel, & Bransford, 2009 23

24. Role-Based Virtual Internship
in Genetic Counseling:
Design-Based Research on Integrated Learning and Assessment Environment
Slide courtesy of the LIFE Center
Phase 1: Preparation
Students engage in self-directed.,
collaborative learning and research to
prepare themselves for the case.
Phase 2: Formative Assessment +
Feedback Students interact with a virtual
Mentor and other experts (e.g., their classroom
teacher) and as needed, to refine their ideas.
Phase 3: Performance / Phase 4: Reflection
Summative Assessment Students reflect on ways to improve
Students counsel their Virtual Clients and both their own learning and the
assessment design. 24
answer their questions.

25. Virtual Genetics Counselor Internship Prototype:
Students know the people in the simulation are not real
but can imagine they are real.
Teacher: What questions do you think the parents will ask?
Student: „Is my child going to have sickle cell or not?‟
Teacher: What is your answer to that question.
Student: „We don't know yet. We don't have enough
information.‟
Teacher: Okay, so picture this. I am these really worried,
potential parents. And you guys are these experts. And you are
telling me you don't know!
Student: Okay. „There is a low chance.‟
Teacher: Like, I am um, 'Low! What are you talking, five? Fifty?
Now I am starting to get worried.' You know what I am saying?
Slide courtesy of the LIFE Center
25
25

26. On Virtual Internship:
”It’s better cause you’re actually interacting
rather than just sitting and listening.”
On their Role:
“It sort of made us feel like we were part of what was going on, like…I don't know, like, like
the actual counselor who was there, we had to go and prepare things and doing quick
research and then putting it all together.”
On Linking School to Career:
“Before this I didn’t really even know
what Genetic Counseling was.”
On Collaboration:
“It puts things in perspective.
It’s almost like looking in
a mirror I guess. When you
talk versus when you read
a question in your head. Like when you say it out loud it makes more sense.”
Slide courtesy of the LIFE Center 26

27. The Challenge Ahead
 We now have the capability to capture complex
skills in the course of assessments that are
actually engaging.
 The main hurdle is abstracting measures that are
both valid and reliable from performance on
these tasks.
- We need evidence that performance on these embedded
assessments predicts what students will do in other contexts.
 The secondary challenge is finding ways to
develop such assessments at much lower cost.
27

28. Diagnoser Assessments: Force and Motion
28

29. Diagnoser Assessments: Force and Motion
29

30. ChemFacets DeBarger & Minstrell
30

31. ChemFacets
31

32. 32

33. 33

34. 34

35. Diagnoser Assessments: Teacher Report
http://www.diagnoser.com
35

36. National & International Use of Technology
in Large-Scale Tests
2010 ??
2009 NAEP
Science
2009 PISA ICTs
(Reading)
2007 Minnesota
State Science Test
2006
PISA
(Science
Pilot)
Source: Jim Pellegrino
36

37. NAEP Hot Air Balloon Assessment Task
37

38. Could we reduce the number of
“drop in from the sky” assessments
by extracting information for
assessment from in-class learning
activities?
38

39. ASSISTments
Mingyu Feng, Neil Heffernan
Worcester Polytechnic Institute
Kenneth Koedinger, Carnegie Mellon
University
39

40. Q1: What should we be assessing
with technology?
40

41. Q2: Should we be assessing
students’ technology skills?
If yes, which skills should we
assess?
41

42. Alternative Perspectives
 Narrow v. broad
 Technical v. cognitive skills
 Discrete v. embedded v. 21st century skills
42

43. ISTE National Educational Technology
Standards for Students (NETS-S)
 Creativity and innovation
 Communication and collaboration
 Research and information fluency
 Critical thinking, problem solving, and
decision making
 Digital citizenship
 Technology operations and concepts
43

44. Partnership for 21st Century Skills
http://www.21stcenturyskills.org 44

45. Source: Robert Kozma, for UNESCO
45

46. NAEP 2012 Technology Literacy
Framework
http://www.edgateway.net/cs/naepsci/print/docs/470
46

47. Comparison of Skill Sets
Skills Partnership Lisbon ISTE ETS PISA NAEP
for 21st Commission NETS iSkills Problem Problem
Century Solving Solving
Creativity/ Innovation X X X
Critical Thinking X X X
Problem Solving X X X X X X
Decision Making X X
Communication X X X X
Collaboration X X X
Information Literacy X X X X
Research & Inquiry X X
Media Literacy X
Digital Literacy X
Digital Citizenship X
ICT Operations & X X X X X
Concepts
Flexibility & X X
Adaptability
Initiative & Self X
Direction
Source: Kozma (2009) Productivity X
Leadership & X
Responsibility
Integrated with X X X
Academic 47
Subjects

48. Assessment & Teaching of 21st Century Skills (ATCS)
Cisco-Intel-Microsoft Project (draft working paper, Binkley et al., Jan. 2010)
 Ways of Thinking
- Creativity & innovation
- Critical thinking, problem solving & decision making
- Learning to learn (metacognition)
 Ways of Working
- Communication
- Collaboration
 Tools for Working
- Information literacy
- ICT literacy
 Living in the World
- Citizenship (local & global)
- Life & career
- Personal & social responsibility (including cultural
awareness & competence)
48

49. Assessment & Teaching of 21st Century Skills (ATCS)
Cisco-Intel-Microsoft Project (draft working papers)
“KSAVE Framework”
- Knowledge
- Skills
- Attitudes/Values/Ethics
 10 X 3 = 30 things to assess (potentially for
each academic subject)
49

50. • Mission: A multi-stakeholder collaboration led by Cisco, Intel & Microsoft to help
transform teaching, learning and assessment of skills needed by students to succeed as citizens
and workers in 21st century.
• Objective: Understanding of specific skill areas
Three skill areas will be clearly defined in terms of what it means for students to grow and
develop in each of
• reasoning within problem solving
• participation in collaborative problem solving
• social networking using technology
Teacher understanding of the developmental approach to learning and assessment within
each of the three new skill definitions.
• Project is based on 3 phases:
Phase 1: Conceptual Phase: white papers (completed)
Phase 2: Construction Phase: Identify 21st century skills to construct assessment tasks
focus on Problem Solving: reasoning skills, collaboration skills, digital literacy and
social networking skills. In addition, teacher understanding of the developmental
approach to learning and assessment within each of the three new skill
definitions. (in progress)

51. Knowledge Building
 Do students move beyond reproducing information to building knowledge,
and is that knowledge cross-disciplinary?
Collaboration
 Do students collaborate with other people and create interdependent work
products?
Use of Technology for Learning
 Do students use ICT in ways that support knowledge building, and do and
learn things that could not be done without ICT?
Problem Solving & Innovation
 Do students solve problems and implement their solutions in the real world?
Self-Regulation
 Does the learning activity have multiple stages, and call on students to plan
their work and assess their work over time?
Skilled Communication
 Do students produce extended communication that is organized around a
central theme and is well developed?
51

52. Approach Being Used in Innovative
Teaching & Learning Research
 Collection and
analysis of samples
of learning activities
and student work as
an alternative to
administering
assessments
52

53. Collection of Instructional Artifacts
in ITL Pilot Countries
 Sample Humanities & Science classrooms of
students 10-15 years of age
 Collect 6 assignments per year from 8 teachers
at each of 6 case study schools
 Collect the work produced by 10 randomly
selected students in response to 4 assignments
per year per class
53

54. Coding Assignments and Work
 Recruit master teachers from
same grade levels and country
(but not same school) as
coders
 Train rubric by rubric using
anchor assignments and
student work for each scale
point
54

55. Train the Trainer Approach
 SRI developing and piloting
rubrics and coding manual
 Face-to-face training of
country evaluators who will
train the teacher coders for
their countries
 Resulting data set sent to
SRI for analysis
55

56. A 21st century learner builds
knowledge
 When students build
knowledge, they move
beyond the reproduction of
information to generate
understandings that are new
to them.
 Students can build
knowledge through
interpretation, analysis,
synthesis, or evaluation.
 We want students to be able
to connect ideas from
different academic
disciplines. 56

57. Knowledge Building
 Knowledge building happens when students combine
new information with what they already know to
generate ideas and understandings that are new to
them.
 Students can do this through interpretation, analysis,
synthesis, or evaluation.
 Knowledge building does not occur when students are
asked to simply reproduce information they have read
or heard from lectures, textbooks, or exposure to the
internet or the media.
57

58. Knowledge Building examples
 Low on Knowledge Building:
- Students write a book report that summarizes the
contents of a book.
 High on Knowledge Building:
- Students research the historical context of a novel
set in 19th century China and use that knowledge as
a lens to interpret the actions of the characters.
58

59. Mr. Sun E. Day News of the Week
59

60. A 21st century learner uses ICT
 Students use technology
tools to support knowledge
building.
 Students do or learn things
that are impractical or
impossible to do without
technology.
 Students shape and create
technology tools to further
their learning and to help
others.
60

61. ICT for Learning examples
 Low on ICT for Learning:
• Students read about the Ebola virus in their textbooks.
 High on ICT for Learning:
• Students identify cases of Ebola by finding news
articles worldwide and plot the spread of the virus
using GIS software.
61

62. ICT Use Rubric for Learning Activities
62

63. ICT Use Rubric for Student Work
63

64. A 21st century learner collaborates
 Students collaborate
when they share
responsibility for
developing a product,
a design, or an answer
to a complex question.
 Students create
interdependent work
products and
negotiate their design
so that each student’s
work is part of a larger
whole.
64

65. Collaboration examples
 Low on Collaboration:
• Students read a chapter from a geometry textbook and
solve the problems at the end of the chapter, writing
alone.
 High on Collaboration:
• Students work in small groups when they use
geometry principles to build a model of their “dream
house.”
65

66. A 21st century learner solves problems
and innovates
 Students solve
problems when they
develop a solution to a
problem that is new or
design a complex
product that meets a
set of requirements.
 Students innovate
when they put a
creative design or
piece of thinking into
practice.
66

67. Problem-Solving & Innovation
examples
 Low on Problem-Solving and Innovation:
• Students follow step-by-step instructions to analyze
the quality of water samples taken from a nearby
stream.
 High on Problem-Solving and Innovation:
• Students propose initiatives to improve water quality
in their region, and send their recommendations to
the local environmental board.
67

68. A 21st century learner regulates his/her
own learning
 Students have the
opportunity to self-regulate
when they work on long-
term projects with multiple
parts.
 Students plan their own
work and monitor their
progress.
 Students can assess the
quality of their own work
when they understand
expectations before the
assignment is completed.
68

69. Self Regulation examples
 Low on Self Regulation:
• Students write about their family during class.
 High on Self Regulation:
• Students work for two weeks on a project about their
cultural heritage, in which they have to schedule and
plan their internet research, a rough draft, a final draft,
and an individual presentation.
69

70. A 21st century learner communicates
skillfully
 Students make an
argument or present a point
of view.
 Students provide sufficient
detail, data, or evidence to
support the argument or
viewpoint.
 Students organize their
communication around a
central theme.
 Students attend to audience
interests and needs.
70

71. Skilled Communication examples
 Low on Skilled Communication:
• Students answer multiple-choice questions about
photosynthesis.
 High on Skilled Communication:
• Students write an explanation of how photosynthesis
works and why it is essential to plant life.
71