In a recent paper co-authored with Sarah Timbs, Glenn Anderson introduced a “simple, non-intrusive and efficient way of measuring English language proficiency that is independent of the commitment of lecturers and tutors, or even the interest level of students” that would “provide a foundation for measuring the impact of ELP strategies as well as encouraging students to take an interest in their ELP from entry to exit”. In this video, Glenn takes a closer look at the incentives of course-wide assessments such as e-Portfolios for student engagement, graduate recruiters, and critically the engagement of academic staff in its implementation.
Lemons, Catastrophic Climate Change, and e-Portfolios: Implementing sustainable course-wide assessments
1.
2. Lemons, Catastrophic Climate
Change, and e-Portfolios:
Implementing sustainable course-wide
assessments
Dr Glenn Anderson and Sarah Timbs
CRICOS Provider Code 00301J
7. Information, Thresholds &
Cooperation
What are the incentives to free-ride?
• Costs of reducing CO2 emissions are born by each country, but the
benefits are shared
• If every country complies with the agreement, the reduction in costs
by any one country defecting would outweigh the reduction in
individual benefit
• There is an incentive to free-ride “Prisoners’ Dilemma”
But wouldn’t breaching 350 ppm result in catastrophic climatic change?
• Any one country breaking the agreement would result in catastrophic
climate change
• Now costs would far outweigh the benefits for each country
• Coordination is a realistic possibility
But the actual threshold is unknown.
• 350 ppm is an average. Is the threshold for irreversible climate change
250 or 450?
• The uncertainty over the threshold reduces the expected impact and
makes free-riding a possibility once more. “Prisoners’ Dilemma”
Barrett, S., & Dannenberg, A. (2014). On the sensitivity of collective action to uncertainty about climate
tipping points. St. Louis: Federal Reserve Bank of St Louis.
8. Information, Thresholds &
Cooperation
• Lemons problem (Akerloff, 1970)
• Two types of 2nd hand cars
• 50% Good quality cars worth $80,000 for the owners
• 50% are poor quality cars (“lemons”) worth $20,000 for the owners
• If consumers pay the average price of $50,000 only poor quality cars
are offered
• Students look to signal high quality (Spence, 1973)
• High achievers pay a premium for a university with a good reputation
• Low achievers not willing to pay the premium (don’t expect to earn
as much)
• Two problems
• Access does not depend on graduate’s expected earnings alone (the
rationale for HECS)
• Link between reputation and quality is “loose” (Podolny, 1993)
9. Two informational
uncertainties undermine
collaboration
Uncertainty about teaching quality
• “Although some academic staff claimed to have been teaching
certain academic literacies, it was difficult to verify such claims..”
(Murray & Nallaya, 2014)
Uncertainty about graduate quality
• “How (do you) know (your) graduates have attained
threshold levels?”(Arkoudis, 2014)
10. Uncertainty: Teaching Quality
“Although some academic staff claimed to have been
teaching certain academic literacies, it was difficult to
verify such claims..” (Murray & Nallaya, 2014)
• I get on well with the language specialists and we have worked
together to redesign the syllabus integrating literacy skills into
learning tasks and assessment of my first year unit
• Everyone in the program are very supportive
• But I’ve become the ‘go-to person’ when there are any issues
with the student literacy
• No-one else is willing to commit and the current arrangement
is unsustainable
11. Uncertainty: Teaching Quality
Why does the agreement break down?
Assume two types of students
• Intrinsically motivated attend literacy workshops
• Most other students are weakly motivated
• Lack intrinsic interest
• Working
Faculty members come together to design a program that will
improve generic skills
• Every hour that an instructor spends on literacy, returns an extra
$50 for the program
• 10 teachers agree to spend 1 hour each
• Cost of an hour for each teacher is $10 (opportunity cost:
research/publications)
12. Uncertainty : Teaching Quality
If everyone cooperates
• Total benefit= $500 & everyone receives $50 each
The last instructor has an incentive to break the agreement
• If the first 9 instructors contribute, payoff per instructor is at least 450/10
=$45.
• If the last instructor contributes: 500/10 =$50 each
• If not, the last instructor receives $45 + $10 = $55, while the others receive
$45
• Total benefit= $450 (i.e., lower)
So the 2nd last also has an incentive to break the agreement
• The 2nd last instructor expects last to break the agreement
• Keep agreement: $45. Defecting yields $40 +$10 = $50. Incentive to defect
• Total benefit $400 (i.e., lower again)
Iterate and no-one has an incentive to keep to the agreement
• Total benefit = $100 & every receives only $10 each
Cannot monitor & enforce agreement “Prisoners’ Dilemma”
13. Uncertainty: Teaching Quality
Bonus if $500 threshold is achieved
• Suppose there is a bonus of $15 for each teacher if the
threshold yielding $500 is achieved (or a penalty of $15 if not
achieved)
• The last teacher. If everyone contributes, the last teacher
receives $50 +15 =65 (bonus for meeting threshold). If the last
teacher does not contribute, when everyone else contributes,
$45 +10 =55.
• If everyone else contributes, it pays for the last teacher to
contribute. If no-one else contributes it pays the teacher not
to contribute
• So as long as the bonus is large enough a prisoners dilemma
game is converted into a coordination game in which
agreements are more easily maintained (Barrett, 2013)
14. Uncertainty: Teaching Quality
Summary so far
• Reputation is based on graduate opportunities. We assumed
reputation is a reliable indicator of quality – graduate competencies.
• However, the contribution of each instructor is not directly
observed. This makes monitoring and enforcement of agreement
problematic prisoners’ dilemma
• Nevertheless, linking a bonus/penalty to achievement/non-
achievement of a threshold standard means there is no incentive for
a teacher to defect if all other instructors are keeping to the
agreement
• Bonus/penalty: when an institution offers students superior/inferior
opportunities via signalling with their reputation
• Competition between HEIs can yield the optimal outcome
16. Uncertainty: Graduate Quality
But concerns remain that competition may not drive quality
improvements
• “To my knowledge very few universities confront the English
language problem by requiring a formal English test and then
mandating supplementary English courses for students who
are deficient. There is widespread recognition of the English
problem. But if a particular university takes unilateral action to
require remedial courses this would add to the fees the
overseas students must pay and put the institution at a
competitive disadvantage relative to the competition.” (Birrell,
2007, p.62-3, emphasis added)
Why wouldn’t offering a competency with strong employer
demand be seen as an opportunity to gain competitive
advantage?
17. Uncertainty: Graduate Quality
Is the link between graduate competencies (hence, teacher effort)
and graduate opportunities reliable?
• Employers look to institutional prestige & grades to improve the
odds of selecting the ‘right candidate’
• institutional reputation, rankings, degree 2:1 or over, entry scores
• Validity & reliability of selection process questionable:
• 75 per cent of vacancies registered with the Graduate Recruitment
Bureau (UK) required 2:1 or above.
• After a 18 month study, Earnst & Young found the 2:1 criteria failed
to predict performance.
• So how will my collaboration with the language specialist impact on
the students graduate opportunities, when I know many employers
will only select from the “top 10” institutions?
• I teach in a “top 10” university. Will my students be any worse off if I
don’t collaborate?
• Any bonus or penalty is far from assured, at least within the tenure
of many academics
18. Uncertainty: Graduate Quality
“How (do you) know (your)
graduates have attained threshold
levels?”(Arkoudis, 2014)
•Climate Change: uncertainty about
the threshold
•Higher Education: uncertainty
about the link between graduate
competencies & graduate
opportunities
19. Uncertainty: Graduate Quality
Link between graduate capabilities & opportunities is
tenuous (Martin, 2011)
“The importance of academic reputations means administrators
and governing boards are narrowly focused on public relations and
they scrupulously avoid controversies. As conditions change, they
do not attempt to reallocate resources; instead, they seek to solve
all campus problems by raising more money. The public relations
mindset adversely selects for lower academic quality.
Administrators divert resources from real quality because those
programs are slow to develop, the benefits are difficult to
measure, and the results are most likely to be realised during the
next president’s tenure. Instead, administrators prefer projects
with immediate public relations value and obvious tangible results,
such as facilities competition and media events.”
Martin, R. E. (2011) The College Cost Disease. Edward Elgar, pviii
20. Collective Action Dilemma
Individualism, unknown teacher quality & unknown graduate
quality ….
“The prevailing norm of academic individualism may therefore
impede the systematic monitoring or measuring of student
achievement that is crucial to the improvement of academic
quality. Without public information about the value-added by an
academic program there are insufficient incentives for individual
faculty members to enter into the coordinated activity necessary to
produce academic programs with the academic coherence and
structure research suggests is associated with student learning. The
improvement of academic quality thereby represents a classic
dilemma of collective action” (Dill & Soo, 2004, pp. 16-17)
… nexus between graduate competencies & opportunities
result in a collective action dilemma
21. Collective Action Dilemma
CASE 1: University of South Australia: An attempt to embedded
academic literacies from above…
Another highly significant challenge that impeded the embedding process,
and which relates to the kind of attitudinal problems discussed above, was
that of compliance among those tasked with implementing it on the ground.
Although it was mandated by the Deputy Vice Chancellor: Teaching and
Learning that the embedding of academic literacies should be implemented
in targeted programmes, there were no clear directives from the university’s
Senior Management Group or the Heads of Schools about the process and
no punitive measures in place should those responsible for implementation
fail to bring about – or attempt to bring about the required change. Given
the scale and ambition of this institution-wide innovation, clear directives
from senior management regarding the significance of undertaking the
embedding process, deadlines for compliance and the consequences of
failure to implement would have provided the project with greater credibility
and momentum. (Murray & Nallaya, 2014, p. 11)
….undermined by lack of cooperation
22. Collective Action Dilemma
Failure to cooperate
• In the event, it was left almost exclusively to the Language and
Learning team within the identified schools to oversee the
embedding process and the impression created was that it was the
sole responsibility of the English language tutors to facilitate that
process. The consequence of this was that Course Coordinators
perceived the initiative as unimportant and unnecessary. This in turn
had a somewhat demoralising effect on language tutors who at
times felt as though they were swimming against the tide and that
attempts to bring about change were futile.
23. CASE 2: Coastal Carolina University: Using badges to assure learning
in an English composition course
• Challenge: “numerous sections of first-year composition courses are
taught by a diverse number of tenured, tenure-track, lecturer and
adjunct faculty; during the current academic year (Fall 2015 - Spring
2016), 4,364 students enrolled in 237 sections of English 101 and
English 102 courses were taught by 79 different faculty members of
varying ranks.”
• “the badges provide a baseline instructional consistency — but not
a standardization — as they ensure learning outcomes are
addressed across all sections. The digital badge program is tied
directly to academic credit, which increased the traditionally three-
credit-hour composition courses by one credit hour. Badges account
for 18 to 24 percent of the final course grade. “
Reid, Alan, and Denise Paster. "A Digital Badge Initiative: Two Years Later -- Campus Technology".
Campustechnology.com. N.p., 2016. Web. 18 Apr. 2016.
Collective Action Dilemma
24. CRICOS Provider Code 00301J
Badges offer an effective measure of learning outcomes….
“However, one thing is certain: This initiative supports
collaboration and pedagogical innovation throughout our
first-year composition program, and it makes visible our
commitment to student learning and success in our
composition courses.”
….allowing the success of individual initiatives to be monitored
A key challenge …
“The digital badge initiative at Coastal Carolina University still faces
some big questions. At the forefront of our minds are philosophical
issues such as how to translate the value of these badges into a real-
world currency for our students ……”
…is converting gains in quality into improved graduate
opportunities
Collective Action Dilemma
25. Pilot Project: Assessing and
Developing English Language
Proficiency
• Good Practice Principles for English language proficiency:
• “Principle 7. Students’ English language development needs
are diagnosed early in their studies and addressed, with
ongoing opportunities for self-assessment.”
• The Higher Education Standards Framework (2011) requires
English language proficiency to be
• “incorporated into course design (Section 1.3) and
assessment” (Section 5.6), as well as
• “supported through entry requirements” (Section 3.2).
CRICOS Provider Code 00301J
26. Pilot project
Issues in implementing embedding
Embedding = Academic staff + learning advisors support
students in specific context; however: issues of time,
commitment and sustainability
“Academics are concerned about their students’
communications skills, but do not believe they have the time
and expertise to address these concerns within the disciplinary
curriculum… (learning) advisers have expertise in developing
student’s academic language and literacy, but developing
collaborative approaches within disciplinary curricula is often hit
and miss”
(Arkoudis, 2014, p. 12).
How can we overcome this impasse?
CRICOS Provider Code 00301J
27. Pilot Project
English language specialists assess existing assignments for
ELP with the results and feedback carried forward to
benchmark future assignments.
• Incentives to achieve threshold standards are enhanced
by language specialists assessing ELP using assignments
from the curriculum
• Sustainability is more likely since ELP assessments are
independent of the commitment of academics (from F/T
to casual)
• Students/staff are able to take a developmental
approach since assessments are administered on an
ongoing basis over the degree
CRICOS Provider Code 00301J
28. Pilot Project
• Conducted at Curtin University Sydney, July to October
2015
• Three selected units + written assignments
• Assessment Tool based on IELTS writing criteria and
simplified into everyday language
• Students used Assessment Tool to self-assess their ELP
• Learning advisor assessed students using the same Tool
• Results were given to students to compare
• Students completed opinion surveys
CRICOS Provider Code 00301J
29. Assessment Tool
Three parts: coherence + cohesion; lexical resource; grammatical
range + accuracy
Student adds total marks, calculates percentage
Example of results:
Basic writer 50-65%
You have a basic command of written English, coping with overall
meaning in most situations, though you are likely to make many
errors. You may be unable to use more complex language. To
improve: read in English as often as you can. Keep lists of
unfamiliar vocabulary and revise them regularly. Practice using
new words in sentences.
CRICOS Provider Code 00301J
Pilot Project
30. Pilot Project
Findings and discussion
• Students assessed themselves reasonably accurately
• Lexical resources were under-estimated
• Grammatical range and accuracy was over-estimated
• The students’ assessments mostly aligned overall with
the learning advisor’s assessments
• 80% of students felt it was important to have their
English language ability assessed in their assignment
• 73.4% of students felt the Assessment Tool was easy to
use and understand
• As for the costs, it is estimated that it takes an hour to
mark four assignments (approx. 2000 words)
CRICOS Provider Code 00301J
31. Further Research
1. Does benchmarking against a previous ELP
assessment improve ELP and assignment outcomes?
• return to students in our current sample and ask
them to repeat the exercise
• compare their average scores and ability to self-
assess with other students
2. What are the challenges, in terms of design,
management and monitoring, for a program-wide
approach to embedding?
3. The role badging and e-Portfolios for supporting
threshold standards and strenthening the link between
graduate competencies & opportunities
CRICOS Provider Code 00301J
32. References
• Akerlof, G. A. (1970). "The Market for 'Lemons': Quality Uncertainty and the Market
Mechanism". Quarterly Journal of Economics (The MIT Press) 84 (3): 488–500
• Arkoudis, S. (2014). Integrating English Language communication Skills into
Disciplinary Curricula: Options and Strategies. Sydney: Office of Learning and
Teaching, Department of Education.
• Arkoudis, S., & Tran, L. (2010). Writing Blah, Blah, Blah: Lecturers’ Approaches and
Challenges in Supporting International Students. International Journal of Teaching and
Learning in Higher Education 2010, 22(2), 169-178.
• Barrett, S., & Dannenberg, A. (2014). On the sensitivity of collective action to uncertainty
about climate tipping points. St. Louis: Federal Reserve Bank of St Louis.
• Campbell, J., & Li, M. (2008). Asian Students’ Voices: An Empirical Study of Asian
Students’ Learning Experiences at a New Zealand University. Journal of Studies in
International Education, 12(4), 375-396.
• Dill, D. D. (2005). Are Public Research Universities Effective Communities of Learning?
The Collective Action Dilemma of Assuring Academic Standards. Paper Presented at
the Symposium: Future of the American Public Research University, 25 February.
CRICOS Provider Code 00301J
33. References
• Department of Education, Employment and Workplace Relations. (2009). Good
Practice Principles for English language proficiency for international students in
Australian universities. Retrieved from
http://www.aall.org.au/sites/default/files/Final_Report-
Good_Practice_Principles2009.pdf
• Harris, A. J., & Ashton, J. (2011). Embedding and integrating language and academic
skills: An innovative approach. Journal of Academic Language and Learning, 5(2), 73-
87.
• Martin, R. E. (2011) The College Cost Disease. Edward Elgar
• Murray, N. & Nallaya, S. (2014): Embedding academic literacies in university
programme curricula: a case study, Studies in Higher Education, 1 – 17.
• Podolny, J. M., 1993. A Status-Based Model of Market Competition. American Journal of
Sociology, pp. 829-872.
• Reid, A. J. & Paster, D.(2016). Digital badges in undergraduate composition courses: Effects
on intrinsic motivation. Journal of Computers in Education, 2(4), 377-398.
• Reid, A., and Paster, D.. "A Digital Badge Initiative: Two Years Later -- Campus
Technology". Campustechnology.com. N.p., 2016. Web. 18 Apr. 2016.
• Spence, M. (1973). "Job Market Signaling". Quarterly Journal of Economics 87 (3):
355–374.
CRICOS Provider Code 00301J
