Dr. Richard Hodson and Joel Hensley covered their ongoing research into the effect of feedback order on student writing. Reporting on their initial findings from a small-scale exploratory study, Hodson and Hensley discussed perceived similarities and differences in learners’ final essays depending on the order in which feedback was given: global revisions first and then local corrections, or vice versa.
1. PEER AND TEACHER FEEDBACK
ON STUDENT WRITING:
WHAT, HOW AND WHEN?
Richard Hodson
Joel Hensley
University of Nagasaki, Siebold
Nagasaki JALT
21 January 2012
2. Background
Two essay writing classes in a Japanese public
university
Fifteen 90-minute classes per semester
Elective, TOEIC requirement: 500
Textbook used:
Ready To Write More
by Blanchard and Root
(Longman)
Difference between classes: order in which
feedback on writing was given
3. Class schedule
“Diagnostic” essay (no local or global check)
Paragraph practice
First essay (“Pressures”)
Classification essay Peer-editing
Cause/effect essay stage
Comparison essay
Problem/solution essay
Final essay (no local or global check)
4. Definitions
Local
Grammar
Vocabulary and word choice
Spelling, capitalization, punctuation
Global
Essay structure
Style
5. Research question
Does the order of feedback in the writing process have
an effect on student writing?
Order of feedback:
Local first (teacher edit), followed by global (peer edit)
vs.
Global first (peer edit), followed by local (teacher edit)
Local editing: done manually by teachers
Global peer-editing: students used checklist of criteria
for each essay
6. Editing process
Class J Class R
First draft First draft
Student rewrites to Student rewrites to
Local check by
Global check by peer include peer incorporate teacher
teacher
suggestions corrections
Second draft Second draft
Student rewrites to Student rewrites to
Local check by
incorporate teacher Global check by peer include peer
teacher
corrections suggestions
Finished essay Finished essay
7. Study participants
Original class size:
J = 13 students
R = 27 students
Selection criteria:
No more than one absence during the semester
Submitted all assignments
Class Grade Female Male Total TOEIC TOEIC
average sd
1 2 3
J 1 2 3 5 1 6 625 134
R -- 10 3 11 -- 11 557 83
8. Texts for analysis
“Diagnostic” essay (no local or global check)
Paragraph practice
First essay (“Pressures”)
Classification essay Peer-editing
Cause/effect essay stage
Comparison essay
Problem/solution essay
Final essay (no local or global check)
9. Final essay topics (1)
Classification
Types of vacation activities
Types of colleges/universities
Types of websites
Cause/Effect
The reasons you decided to major in English/come to this
university
The effects that the extension of the Shinkansen to Nagasaki
might have
The effects that the Internet has had
10. Final essay topics (2)
Compare/Contrast
An aspect of Japanese culture and of the culture of another
country
Japan now and Japan in another time period
Two different products on sale in Japan
Problem/Solution
The falling birth rate in Japan
People getting music, movies and TV shows illegally from the
Internet
Bullying in schools
11. Analytical measures
Local
Average sentence length
Gunning-Fog readability
“a weighted average of the number of words per sentence,
and the number of long words per word”
% of error-free sentences (average of J & R scores)
Global
12 criteria
J & R agreed score
12. Global criteria (1)
Introduction
Is there enough background information?
Has the author used at least one intro
technique (anecdote, etc.)?
Thesis Statement:
Has the subject been given?
Has the focus (main ideas) been provided?
Is there a clear purpose (not necessarily in
the TS itself)?
13. Global criteria (2)
Body
Does every body paragraph contain a Topic
Sentence with topic and focus?
Does every body paragraph provide adequate
support to develop and prove the thesis?
Do the body paragraphs achieve unity (no
irrelevant, vague, or repetitive sentences)?
Are the body paragraphs arranged in a logical
order?
14. Global criteria (3)
Conclusion
Has the Thesis Statement been summarized or
restated in different words?
Has the author used at least one conclusion
technique (question, etc.)?
Other
Has the essay been formatted correctly?
15. Raw comparison
local
percentage of sentences
correct average sentence length Gunning-Fog readability
Class R Class J Class R Class J Class R Class J
diagnostic 29.61% 29.52% 11.72 12.23 7.72 8.19
pressures 59.86% 68.90% 13.19 15.24 7.68 8.55
causes 54.67% 60.19% 13.82 13.24 8.37 7.69
compare 63.02% 58.49% 13.05 13.41 7.79 8.21
final 25.72% 29.73% 13.35 14.32 8.21 8.44
global
Class R Class J
diagnostic 6.09 7.00
pressures 9.27 9.83
causes 9.64 9.83
compare 10.73 10.50
final 10.27 9.50
16. Results: local (by class) 1
1.5
Percentage of sentences correct (Z-score)
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
-1.5
17. Results: local (by class) 2
2
Average sentence length (Z-score)
1.5
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
-1.5
-2
18. Results: local (by class) 3
1.5
Gunning-Fog readability (Z-score)
1
0.5
0
diagnostic pressures causes compare final Class R (local first)
Class J (global first)
-0.5
-1
-1.5
-2
19. Results: local (by class) 4
1.5
Collective local variable (Z-score)
1
0.5
Class R (local first)
0
Class J (global first)
diagnostic pressures causes compare final
-0.5
-1
•Impact of starting proficiency?
•Final essay switch
-1.5
20. Results: global (by class)
1 Global criteria (Z-score)
0.5
0
diagnostic pressures causes compare final
Class R (local first)
-0.5
Class J (global first)
-1
-1.5
•Substantial improvement in both classes
•Final essay switch
-2
21. Results: local (by TOEIC)
1.5
Collective local variable (Z-score)
1
0.5
High TOEIC score
0
Low TOEIC score
diagnostic pressures causes compare final
-0.5
-1
-1.5
22. Results: global (by TOEIC)
1
Global criteria (Z-score)
0.5
0
diagnostic pressures causes compare final
High TOEIC score
-0.5
Low TOEIC score
-1
-1.5
-2
23. Results: local (by TOEIC)
80
Collective local variable (raw score)
70
60
50
40 High TOEIC score
Low TOEIC score
30
20
10
0
diagnostic pressures causes compare final
24. Results: global (by TOEIC)
Global criteria (raw score)
12
10
8
6 High TOEIC score
Low TOEIC score
4
2
0
diagnostic pressures causes compare final
25. Discussion (1)
Does the order of feedback in the writing process
have an effect on student writing?
Both classes made substantial 1
0.5
diagnostic-final global 0
-0.5
Class R (local
first)
improvement -1
Class J (global
first)
-1.5
-2
Local improvement?
1.5
Less
substantial diagnostic-final 1
More variation in peer-editing stage 0.5 Class R (local
first)
0
Between classes -0.5
Class J (global
first)
Between different essay types -1
Amongst local variables -1.5
26. Discussion (2)
Class J outperformed Class R in global criteria
during peer-editing stage
Class R outperformed Class J in local criteria during
peer-editing stage
Hypothesis
During the peer-editing stage, classes showed
greater improvement in the area focused on first
[global/local].
27. Discussion(3)
Class R outperformed Class J globally in final essay
Class J outperformed Class R locally in final essay on
2/3 of criteria analyzed
Hypothesis
During the peer-editing stage, focusing on global
feedback first allowed students to make more immediate
improvement on global criteria. However, in post-peer-
editing [final essay], students focused more on the
last/most recent feedback stage [local/global] when
completing their essay independently.
28. Limitations of study
Sample size
17 students
85 essays (36,636 words)
Analytical measures
Three local variables only
Inter-rater reliability issues
Nature and efficiency of feedback
Teacher (local)
Peers (global)
Extent of student adoption/use of feedback
During the semester
When writing final essay
29. Future research possibilities
Further analysis of existing sample
Include more
texts
Improve number & accuracy of local variables
Continue existing study with larger sample
Analyze nature, quality, and extent of adoption and
efficiency of feedback
30. Ongoing research
Continue existing study with larger sample
Class J: 21 students
Class R: 12 students
Possibly data from a further 14 students?
Standardized local feedback:
ETS Criterion® Online Writing Evaluation Service
32. Ongoing research issues (2)
Data-gathering v. pedagogy
Difficultto replicate experimental conditions in
dynamic classrooms
Possible need to change final essay may invalidate
comparison between groups?
Broader question:
Does our working hypothesis suggest a change to
pedagogical practice?
If so, should classroom practice be changed, at the risk
of invalidating further research?