This presentation provides an overview of fairness in general, and specifically, how it can be applied to code reviews. It also describes an empirical study of openstack that shows that fairness in their code review process requires some attention as some participants feel that their contributions are being unfairly reviewed.

1. “Was my contribution fairly
reviewed”
A framework to study the perception of fairness in
modern code reviews
Daniel M German
University of Victoria
dmg@uvic.ca

3. http://github.com/dmgerman/papers

4. What does it mean to be fair?

6. Is ICSE fair?

7. 62% of reviews were considered to be fair (2017)
Is ICSE fair?

8. 62% of reviews were considered to be fair (2017)
48% (for rejected papers)
Is ICSE fair?

9. ICSE experiments

10. ICSE experiments
• 2016 and before:

11. ICSE experiments
• 2016 and before:
• Introduction of a Program Board

12. ICSE experiments
• 2016 and before:
• Introduction of a Program Board
• Rebuttal period

13. ICSE experiments:

14. ICSE experiments:
• 2017

15. ICSE experiments:
• 2017
• Structured reviews

16. ICSE experiments:
• 2017
• Structured reviews
• Reviewers were anonymous among PC

17. ICSE experiments:
• 2017
• Structured reviews
• Reviewers were anonymous among PC
• Public “Reviewer guidelines”

18. ICSE experiments:
• 2017
• Structured reviews
• Reviewers were anonymous among PC
• Public “Reviewer guidelines”
• No “subreviewer” (e.g reviews done by students)

19. ICSE experiments:
• 2017
• Structured reviews
• Reviewers were anonymous among PC
• Public “Reviewer guidelines”
• No “subreviewer” (e.g reviews done by students)
• No list of research topics

20. ICSE experiments

21. ICSE experiments
• 2018

22. ICSE experiments
• 2018
• Gender balance of PC

23. ICSE experiments
• 2018
• Gender balance of PC
• Double blind

24. ICSE experiments
• 2018
• Gender balance of PC
• Double blind
• Authors received numerical scores

25. What is ICSE trying to solve?

26. What is ICSE trying to solve?
• Inconsistency of reviewing

27. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias

28. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias
• Topic

29. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias
• Topic
• Interpersonal

30. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias
• Topic
• Interpersonal
• Lack of transparency

31. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias
• Topic
• Interpersonal
• Lack of transparency
The ICSE reviewer lottery

32. What is ICSE trying to solve?
• Inconsistency of reviewing
• Bias
• Topic
• Interpersonal
• Lack of transparency
The ICSE reviewer lottery
Some participants consider the process unfair

33. Participants are likely to leave a
system if they perceive it to be
unfair

34. What does it mean to be fair?

35. What does it mean to be
treated fairly?

36. Fairness

37. Fairness
• Also called “Justice”

38. Fairness
• Also called “Justice”
• Concept from:
• Philosophy/Law/Psychology/
Sociology/Management/
Economics…

39. Fair (Merriam Webster):

40. Fair (Merriam Webster):
1.

41. Fair (Merriam Webster):
1.
a. Marked with impartiality and
honesty: free from self-interest,
prejudice, or favouritism

42. Fair (Merriam Webster):
1.
a. Marked with impartiality and
honesty: free from self-interest,
prejudice, or favouritism
b. Conforming with the established
rules

43. Justice (Merriam-Webster):

44. Justice (Merriam-Webster):
1.

45. Justice (Merriam-Webster):
1.
a. The maintenance or administration
of what is just, specially by the
impartial adjustment of conflicting
claims or the assignment of
merited rewards or punishments

46. Justice (Merriam-Webster):

47. Justice (Merriam-Webster):
2.

48. Justice (Merriam-Webster):
2.
a. The quality of being just,
impartial or fair

49. Organizational Fairness/Justice:

50. Organizational Fairness/Justice:
‣ how to decide and allocate

51. Organizational Fairness/Justice:
‣ how to decide and allocate
‣ resources and decisions

52. Organizational Fairness/Justice:
‣ how to decide and allocate
‣ resources and decisions
‣ in a way that is fair and just

53. Organizational Fairness/Justice:
‣ how to decide and allocate
‣ resources and decisions
‣ in a way that is fair and just
‣ to the individuals and the group as a
whole.

54. In general, we care about fairness
when somebody makes a decision
that affects us.

55. Fairness theory “focuses on explaining
when authorities should be held
accountable for unfavourable events”

56. How to increase the
perception of fairness?

59. Distributive Fairness

60. Distributive Fairness
Outcomes are allotted

61. Distributive Fairness
Outcomes are allotted
to persons

62. Distributive Fairness
Outcomes are allotted
to persons
whose relative shares

63. Distributive Fairness
Outcomes are allotted
to persons
whose relative shares
can be described by some functional rule

64. Distributive Fairness
Outcomes are allotted
to persons
whose relative shares
can be described by some functional rule
and some standard

65. Distributive Fairness

66. Distributive Fairness
• Should those that contribute more
receive more?

67. Distributive Fairness
• Should those that contribute more
receive more?
• How do you measure contribution?

68. Distributive Fairness
• Should those that contribute more
receive more?
• How do you measure contribution?
• Is there a minimal standard defined in
the system?

69. Equity vs Equality

70. Equity vs Equality
• Equity:
• Those that contribute more should
receive more

71. Equity vs Equality
• Equity:
• Those that contribute more should
receive more
• Equality:
• Everybody deserves the same

72. Organizations favour equity

73. Organizations favour equity
• Rewards those that contribute more
• Improves performance

74. Newcomers are special case

75. Newcomers are special case
• Ok to have a positive bias to
newcomers
• They have not yet contributed

76. Procedural Fairness

77. Procedural Fairness
• Concerned with the rules/process of making a decision:

78. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time

79. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time
• Bias suppression of decision makers

80. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time
• Bias suppression of decision makers
• Accuracy of information used in decisions

81. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time
• Bias suppression of decision makers
• Accuracy of information used in decisions
• Control: those affected should have a voice

82. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time
• Bias suppression of decision makers
• Accuracy of information used in decisions
• Control: those affected should have a voice
• Correctability: there should exist an appeal process

83. Procedural Fairness
• Concerned with the rules/process of making a decision:
• Consistency of decisions across time
• Bias suppression of decision makers
• Accuracy of information used in decisions
• Control: those affected should have a voice
• Correctability: there should exist an appeal process
• Ethicality: the process upholds personal and organizational
standards

84. Interactional fairness

85. Interactional fairness
• Individuals are not only concerned with outcomes but how
they are treated

86. Interactional fairness
• Individuals are not only concerned with outcomes but how
they are treated
• Respect: actions and signals that reflect the intrinsic value
that a person has for another

87. Interactional fairness
• Individuals are not only concerned with outcomes but how
they are treated
• Respect: actions and signals that reflect the intrinsic value
that a person has for another
• No rudeness/berating/sexism/racism

88. Interactional fairness
• Individuals are not only concerned with outcomes but how
they are treated
• Respect: actions and signals that reflect the intrinsic value
that a person has for another
• No rudeness/berating/sexism/racism
• Invasion of privacy: no disclosure of secrets/improper
questions

89. Interactional fairness
• Individuals are not only concerned with outcomes but how
they are treated
• Respect: actions and signals that reflect the intrinsic value
that a person has for another
• No rudeness/berating/sexism/racism
• Invasion of privacy: no disclosure of secrets/improper
questions
• Psychological safety: no increase of stress

90. Informational fairness

91. Informational fairness
• What information is used/exchanged is important

92. Informational fairness
• What information is used/exchanged is important
• Truthfulness:

93. Informational fairness
• What information is used/exchanged is important
• Truthfulness:
• Avoid deception: people don’t like to be “lied to”

94. Informational fairness
• What information is used/exchanged is important
• Truthfulness:
• Avoid deception: people don’t like to be “lied to”
• Adequacy of explanations:

95. Informational fairness
• What information is used/exchanged is important
• Truthfulness:
• Avoid deception: people don’t like to be “lied to”
• Adequacy of explanations:
• Reasonable explanations help accepting an
outcome

96. Code Reviews

97. Code Reviews
• A decision process that involves
individuals

98. Code Reviews
• A decision process that involves
individuals
• Prone to fairness concerns

100. Distributive Fairness

101. Distributive Fairness
• Two concerns:

102. Distributive Fairness
• Two concerns:
• Patch level

103. Distributive Fairness
• Two concerns:
• Patch level
• Individual level

104. Patches

105. Patches
• Equity: Some patches

106. Patches
• Equity: Some patches
• are more important than others

107. Patches
• Equity: Some patches
• are more important than others
• require more than others

108. Patches
• Equity: Some patches
• are more important than others
• require more than others
• Need: what is the minimal treatment
that a patch should receive?

109. Distributive Fairness: individuals

110. Distributive Fairness: individuals
• Equity vs Equality

111. Distributive Fairness: individuals
• Equity vs Equality
• Should core developers receive
preferential treatment?

112. Distributive Fairness: individuals
• Equity vs Equality
• Should core developers receive
preferential treatment?
• Newcomers:

113. Distributive Fairness: individuals
• Equity vs Equality
• Should core developers receive
preferential treatment?
• Newcomers:
• What preferential treatment should they
get?

114. Procedural Fairness

115. Consistency and bias suppression

116. Consistency and bias suppression
• Define:

117. Consistency and bias suppression
• Define:
• standards of quality and minimal
requirements for approved patches

118. Consistency and bias suppression
• Define:
• standards of quality and minimal
requirements for approved patches
• prioritization strategies

119. Control and Inf. Accuracy

120. Control and Inf. Accuracy
• Allow authors and non-reviewers to
comment on patches under review

121. Correctability

122. Correctability
• Create an appeal process for
rejected patches

123. Ethicallity

124. Ethicallity
• Define ethical standards on how
code reviews should be done

125. Ethicallity
• Define ethical standards on how
code reviews should be done
• What is Conflict of Interest?

126. Ethicallity
• Define ethical standards on how
code reviews should be done
• What is Conflict of Interest?
• Is self-reviewing allowed?

127. Interactional Fairness

128. Interactional Fairness
• Define a code of conduct

129. Equity vs Equality

130. Equity vs Equality
• A code of conduct seems to imply
that everybody is equal

131. Equity vs Equality
• A code of conduct seems to imply
that everybody is equal
• But in reality, some people are more
important than others

132. Informational Fairness

133. Truthfulness

134. Truthfulness
• Do code reviews with full ongoing
transparency

135. Adequacy

136. Adequacy
• Reviewers should:

137. Adequacy
• Reviewers should:
• provide good feedback

138. Adequacy
• Reviewers should:
• provide good feedback
• explain delays, and

139. Adequacy
• Reviewers should:
• provide good feedback
• explain delays, and
• unexpected treatments.

140. Empirical Study

141. OpenStack

142. OpenStack
• Ecosystem of 30-something projects

143. OpenStack
• Ecosystem of 30-something projects
• Industrial

144. OpenStack
• Ecosystem of 30-something projects
• Industrial
• Most contributions are by paid developers

145. OpenStack
• Ecosystem of 30-something projects
• Industrial
• Most contributions are by paid developers
• 0.5 M SLOCs by 60k contributors

146. OpenStack
• Ecosystem of 30-something projects
• Industrial
• Most contributions are by paid developers
• 0.5 M SLOCs by 60k contributors
• Using code reviews

147. Survey

148. Survey
• 2870 invitations

149. Survey
• 2870 invitations
• 7.4% response rate (213)

150. Have your contributions being treated unfairly?
0
10
20
30
40
Never Rarely Occassionally Often Always

151. According to your experience as a reviewer, do
you perform code reviews unfairly?
0
15
30
45
60
Never Rarely Occassionally Often Always

152. How do you prioritize reviews?

153. Open ended answers

154. Open ended answers

155. Open ended answers

156. Some open-ended answers

157. Equity vs Equality
“A submitter is presented with the
illusion that all submissions are equal,
when in reality the community prioritizes
[code reviews] from known submitters”

158. Newcomers
“I have only tried to make one
contribution. I was so discouraged by
the response from the reviewer I have
not tried again.”

159. Consistency
“I have contributed and reviewed in 3
projects: [A, B] and [C]. I found that [A]
and [B] to be quite reasonable and fair.
As stated above, [C] is a disaster.”

160. Control and Correctability
“[A] core reviewer gave me a -1 which
was not understandable. I asked 4 time
on IRC to discuss it and he ignored
me”

161. Ethicallity
“People look out for friends instead of
looking out for their projects”

162. Interactional Fairness
“Contributions are ignored unless I beg
for attention. I might have to contribute a
minor improvement. I never have time
to beg for attention. If you don’t want
my help, I got the message loud and
clear”

163. OpenStack: Summary

164. OpenStack: Summary
• Overall, it seems to be perceived as
fair

165. OpenStack: Summary
• Overall, it seems to be perceived as
fair
• There are signs of discontent,
specially in some sub-projects

166. Takeaway message

167. Takeaway message
Fairness is an issue that needs
attention in code reviews

168. Takeaway message
Fairness is an issue that needs
attention in code reviews
This framework helps understand what
are the issues and how to address
them

169. To summarize

178. http://turingmachine.org
http://github.com/dmgerman/papers