This document summarizes a pilot study examining how students improve their writing over multiple essays through peer and instructor feedback. The study tracked feedback on 10 essays from 13 students. Both peer and instructor feedback improved linearly over the essays. Peer feedback ratings increased more slowly than instructor ratings. Positive instructor feedback on argument strength and style/mechanics correlated with improved essay quality. More analysis is needed to understand how students apply feedback to different essays and whether the quality of peer feedback improves over time with more sessions. The pilot showed this type of longitudinal study is feasible with a larger sample size.

1. Tracking a Dose-Response Curve
in Peer Feedback on Writing
A Work in Progress
Christina Hendricks
Co-Investigator: Jeremy Biesanz
University of British Columbia-Vancouver
SoTL Symposium, November 2015
Slides available here: http://is.gd/PFBwriting2015
Slides licensed CC-BY 4.0

2. Literature on peer feedback
Receiving peer
feedback improves
writing
(Paulus, 1999; Cho & Schunn,
2007; Cho & MacArthur, 2010;
Crossman & Kite, 2012)
Giving peer feedback
improves writing
(Cho & Cho, 2011; Li, Liu &
Steckelberg, 2010)

3. GAPS:
Most studies look at revisions to a single
essay, not changes across different essays
Draft 1 Draft 2 Draft 3
Essay 1 Essay 2 Essay 3 Essay 4 Essay …n
PFB
PF
B
PF
B
PFB PF
B
PFB
Few studies look at “dose-response curve”

4. Pilot study research questions
1. How do students use peer comments given and
received for improving different essays rather than
drafts of the same essay?
1. Are students more likely to use peer comments
given and received for improving their writing
after more than one or two peer feedback
sessions? How many sessions are optimal?
2. Does the quality of peer comments improve over
time?

5. • Interdisciplinary, full year course for first-years
• 18 credits (6 each in 1st year English, History,
Philosophy)
• Students write 10-12 essays (1500-2000
words)
• Peer feedback tutorials every week (4 stdnts)
http://artsone.arts.ubc.ca
Toni Morrison, Wikimedia Commons,
licensed CC BY-SA 2.0
Osamu Tezuka, public domain
on Wikimedia Commons
Jane Austen, public domain on
Wikimedia Commons
Friedrich Nietzsche, public
domain, Wikimedia Commons

6. Data for pilot study 2013-2014
• 10 essays by 13 participants (130 essays)
• Comments by students in tutorial group (4 in
group) on all essays (n=1219)
• Comments by instructor on all essays
(n=3331)
• All essays and comments coded according to
a common rubric

7. Coding Rubric
Categories
(plus
subcategories, for
11 options)
• Strength of argument
• Organization
• Insight
• Style & Mechanics
Numerical
value
1: Significant problem
2: Moderate problem
3: Positive comment/praise
E.g., STREV 2: could use more textual
evidence to support your claims

8. Progress so far
All student and instructor comments coded
60 of 130 essays
coded
-- (10 essays
each by 6 of the
13 participants)

9. Inter-coder reliability
Fleiss’ Kappa Intra-class
correlation
Student
comments
(n=141)
All categories: 0.61 (moderate)
Most used categories: 0.8
(excellent)
0.96
(excellent)
Essays (n=60) 0.68
(adequate)
3 coders:
• Daniel Munro & Kosta Prodanovich
(undergrads, former Arts One)
• Jessica Stewart (author, editor)

10. LOOKING AT TRENDS IN
COMMENTS OVER TIME

11. 0 2 4 6 8 10 12
024681012
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
INSTRUCTOR
Comments
-
.28**Strength
Style
Organiz.
Insight
-.04*
Number of 2 comments in each categ.

12. 0 2 4 6 8 10 12
01234
Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
STUDENT
comments
Strength
Style
Organiz.
Insight
-.16**
Number of 2 comments in each categ.

13. 0 2 4 6 8 10 12
012345
Essay Number
InstructorNumberofComments
Argument Strength
Style
Insight
Organization
INSTRUCTOR
Comments .31***
Strength
Style
Organiz.
Insight
.08**
.19**
.11**
Number of 3 comments
in each category

14. 0 2 4 6 8 10 12
0.00.51.01.52.02.53.0
Essay Number
StudentNumberofComments
Argument Strength
Style
Insight
Organization
STUDENT
Comments
Strength
Style
Organiz.
Insight
Number of 3 comments in each categ.

15. Student/instructor
agreement on comments
• Average numerical ratings for comments
across all categories agree strongly
between student and instructor (.48****)
• However, this agreement
goes down across essays (-.04**)
o This is because student ratings increase
over time at only half the rate that instructor
ratings do
*p < .05, **p< .01, ***p< .001, ****p < .0001

16. HOW DOES ESSAY QUALITY
CHANGE OVER TIME?
From this slide onwards, we are looking only at
60 essays coded so far, out of the set of 130
(essays from 6 of the 13 participants)

17. Essay
quality
improves
linearly
in 60 essays
0 2 4 6 8 10
3.03.54.04.55.05.5
Essay
EssayQualityRating
Mean essay
quality rating
4.12 out of 7,
SD = .62

18. MORE COMPLEX
ANALYSES

19. Cross-lagged panel design with
auto-regressive structure
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N

20. Path A: Student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects:
• Ratings of 2 in Insight (-.53*)
• Ratings of 3 in Strength (.25*)

21. Path C: instructor comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects
• Ratings of “2” in Strength (-.31**)
• Ratings of “3” in Strength (.51***) and Style/Mechanics
(.34**)

22. Path C: student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects
• Ratings of “2” in Style (.28*) and Insight (.15*)
• Ratings of “3” in Strength (.30*)

23. Path D: Student comments
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N
Significant effects:
• Ratings of 2 in Organization (.16*)
• Ratings of 3 in Style (.16*)

24. Research question 2
How do students use peer comments given
and received for improving different essays
rather than drafts of the same essay?
o Not enough evidence yet to say much about
path D
o Haven’t yet looked at differences in
comments given vs. received

25. Research question 2
Are students more likely to use peer
comments given and received for improving
their writing after more than one or two peer
feedback sessions? How many sessions are
optimal?
o No evidence yet that there is any change
over time in path D

26. Research question 3
Does the quality of peer comments improve
over time?
o No evidence yet that there is any change over
time in path A
Essay Quality
Time 1
Essay Quality
Time 2
Comments
Time 1
Comments
Time 2
B
A
C
D
E
… N
… N

27. Some conclusions so far
• Pilot study: is this sort of study feasible for
larger sample?
o Yes, but probably more so if instructors code
essay quality rather than coders
• Facilitating easy collection of student &
instructor comments is difficult

28. References
• Cho, K., & MacArthur, C. (2010). Student revision with peer
and expert reviewing, Learning and Instruction. 20, 328-338.
• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving
comments. Instructional Science, 39, 629-643.
• Cho, K. & Schunn, C. D. (2007). Scaffolded writing and
rewriting in the discipline: A web-based reciprocal peer review
system. Computers & Education, 48, 409–426
• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved
writing among students through directed peer review, Active
Learning in Higher Education, 13, 219-229.
• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or
assessee: How student learning improves by giving and
receiving peer feedback. British Journal of Educational
Technology, 41(3), 525–536.
• Paulus, T. M. (1999). The effect of peer and teacher feedback
on student writing. Journal of Second Language Writing, 8,
265-289.

29. Thank you!
Christina Hendricks
University of British Columbia-Vancouver
Website: http://blogs.ubc.ca/christinahendricks
Blog: http://blogs.ubc.ca/chendricks
Twitter: @clhendricksbc
Slides available: http://is.gd/PFBwriting2015
Slides licensed CC-BY 4.0