Eaquals Training for Excellence: Assessment, Elaine Boyd

Evaluating and designing
language assessments
Elaine Boyd
University College London
elaine.boyd@ucl.ac.uk
©Eaquals Eaquals – Training for Excellence | Malta | 17-18 November 2017 www.eaquals.org 1

Objectives
• testing principles
• types of assessment
• applying principles
• mapping levels of assessments
• writing assessment tasks
• judging performance
• individual challenges
©Eaquals www.eaquals.org 2Eaquals – Training for Excellence | Malta | 17-18 November 2017

Friday
• Why are principles important?
• What are the principles of assessment?
• What are we measuring?
• How do principles apply to your tasks?
• How do we level tasks?
©Eaquals www.eaquals.org 3
Eaquals – Training for Excellence | Malta | 17-18 November 2017

Why are principles
important?
• Ethics
• Fairness
• Consequences
• Fit for purpose

What are the principles
of assessment?
VALIDITY
• Is the test testing what it claims
to be testing?
• Does the test represent the test
taker’s real life language use?
• Does the test represent learning
activities?

of assessment?
RELIABILITY
• Is every version of the test the
same?
• Is the test the same across time?
• Is every test taker’s performance
assessed in the same way?
How do you check?

of assessment?
PRACTICALITY
• time to deliver
• complexity of instructions
• time to assess
• resources needed
What can go wrong?

of assessment?
IMPACT
• What effect does the test have on
teaching?
• What effect does the test have on
the learner?
• How wide is the impact of the
test?
When do you design for impact?

Principles of assessment
Interlocking or is there a priority?
What is the effect of getting one
wrong?
Who benefits?

Communicative competences (Bachman & Palmer, 1997)
What do we want to
measure?

Example 2: speaking
Format: test taker & test taker Level A2 + B1
Length: 5 - 7 minutes
Candidate A: You organize and trip with an English friend. You make
suggestions and ask your partner if he/she agrees.
Candidate B: You organize and trip with an English friend. You make
suggestions and ask your partner if he/she agrees.

Example 3: writing
Level A2
A Write a description of your best friend. Say:
- what she looks like
- what she likes
- why you like her
B Write a description on a visit to the zoo.
Say:
- what the zoo was like
- what animals you like best and why

How do principles apply
to your tasks?
• What is the purpose of your test?
• Which principle is more evident in your test?
• Which principle is the most relevant for your
context? Why?
• What feedback do you give from the test?
• What would you change? Why?

How do we level tasks?
• CEFR
• Syllabus in Course Book
• School curriculum
• Other validated tasks
• Collaboration

Level: texts & tasks
• What level would you test with this
text?
• Which competences would you test?
• What kind of task would you use to
elicit those competences?
• Could you adapt level? How? Risks?

Saturday
• Rules for writing assessment tasks
• How do we judge performance?
• Issues in formative assessment
• What are your individual challenges?

Writing objective tasks
Multiple choice questions
What is the speaker’s opinion?
a) He supports the project.
b) He does not support the project.
c) He thinks they need to do more work on the project.

Writing objective tasks:
Rules for writing M/C
§ Item must be appropriate level.
§ Question must be clear on its own.
§ Question must identify where to listen or read.
§ Options must capture genuine distraction.
§ Options must be similar length and structure.
§ Options all have the same focus e.g. positive or
negative.
§ Key must be true and only the key is true.
§ Distractors must be plausible.
§ Must test language knowledge not general
knowledge.

Writing objective tasks
Write TWO 3-option M/C items to test
understanding of the listening text.
People prefer nuclear families because 
a. there is more freedom. 
b. there are fewer members. 
c. it is the latest trend in many countries.
2. According to the speaker, young people don’t listen to elders
because they
a. don’t want to be restricted. 
b. think they know better. 
c. learn from their mistakes.

Writing productive tasks
You are teaching a class of A2 level students. The
students are 10 years old.
Write a writing task for these students.
The topic must be Free Time.

How do we judge
performance?
• How do we decide WHAT to grade?
• How do we decide HOW to grade?
• How do we decide on balance/weighting?
• What is the ‘judgement’ difference between
summative and formative tests?
• Can we use criteria as Objectives? How?

Write assessment criteria
Write criteria for Writing task:
• use competences & CEFR
• decide categories
• write top performance descriptor
• decide scale with rationale
• decide weighting with rationale

Formative assessment
• What are the different ways we
can do this?
• What are the challenges to
validity?

Your tests
• What challenges do you face in
your own interaction in tests?
• Can you use any of the principles
to help you?
• What processes could you set up to
resolve?
• What checks could you set up to
review?
• What remains?

Fairness Check
• What am I trying to test?
• How do I know that I’m testing what I think I’m
testing?
• Does the test task reflect the real world and/or
classroom practice?
• Can I replicate this reliably?
• What is are reasonable conditions for the test?
• What will happen when someone fails?
• What effect will this test task have on classroom
practice?

Remember ….....
vYOU ARE ALWAYS MEASURING
POSITIVE ACHIEVEMENT
vNEVER DO IT ALONE!

References
• Fulcher, G. (2010). Practical language testing. London: Hodder.
• Bachman, L., & Palmer, A. (2010). Language assessment in practice. Oxford: Oxford
University Press.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp.
13-104). New York, NY: American Council on education and Macmillan.
• International Language Testing Association (ILTA) guidelines:
https://moodle.ucl.ac.uk/pluginfile.php/4227841/mod_resource/content/1/Week%209
%20-%20ILTA%20guildelines%20for%20practice.pdf
• Dianne Wall (2012) Examining washback: What do we know and what is there left to
explore? http://www.alte.org/attachments/pdfs/files/dianne_wall_jl7so.pdf
• Language Testing site has expert videos, podcasts, articles and references:
• http://languagetesting.info
• Manual for Language Test Development and Examining. ALTE Council of Europe.
2011
• http://www.coe.int/t/dg4/linguistic/ManualLanguageTest-Alte2011_EN.pdf
• EALTA good practice:
http://www.ealta.eu.org/documents/archive/guidelines/English.pdf
• Council of Europe (2009) Relating Language Examinations to the Common European
Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR). A
Manual. https://rm.coe.int/1680667a2d

Eaquals Training for Excellence: Assessment, Elaine Boyd

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Eaquals Training for Excellence: Assessment, Elaine Boyd

Similar to Eaquals Training for Excellence: Assessment, Elaine Boyd (20)

More from eaquals

More from eaquals (20)

Recently uploaded

Recently uploaded (20)

Eaquals Training for Excellence: Assessment, Elaine Boyd