Presentation of Joshua Nicholson from The Winnower on the OpenAIRE workshop "Open Peer Review: Models, Benefits and Limitations"

2. Grey Literature & Social Media
as Open Review

3. social media

4. grey literature

5. social media-
applications that
enable users to
create and share
content or to
participate in social
networking

6. grey literature-
report or
manuscript
circulated or
published by
unconventional
routes

7. how big are these two
entities and how are they
used?

8. reddit: “the front page of the internet”
2,200,000 Unique Pageviews per month
4,900,000 Total Pageviews per month
~100,000-125,000 comments per month.
In r/science
alone}

10. Twitter: “yours
to discover”
320 million monthly
active users worldwide
500 million tweets
(140 character-long
messages) per day
Van Noorden, Nature 2014

12. Facebook:
“Connect
with friends
and the world
around you.”
1.65 billion monthly
active users
worldwide
Van Noorden, Nature 2014

14. Academia.edu
33M members
6M active users
ResearchGate
9M Members

15. Pubpeer

16. Journal clubs

17. Journal clubs
DOI: 10.1111/bcp.12992

18. Blogs

19. Blogs

20. “after 14 months
of informal post-publication
discussion, the hypothesis was
refuted.”
-Yeo et. al, 2016

21. How good is classical pre-
publication peer review?

22. Editors at the British Medical Journal found on
average that only 2 out of 9 major artificial errors
were detected
(Schroter et al., 2008)

23. At JAMA only 2 out of 8 artificial errors were
noticed
(Godlee et al., 1998)

24. At the Annals of Emergency Medicine, 68% of
reviewers did not realize the conclusions were not
supported by the evidence
(Baxt et al., 1998)

25. Journals rejected previously accepted/published work 8
out of 9 times when names and institutions of authors
were changed from prestigious universities (Harvard,
Stanford) to fake names like “Tri-valley Institute.”
(Ceci & Peters, 1982)
footnote: one author was denied after
performing this study

26. Agreement between reviewers as to whether
papers should be accepted, revised or rejected is
no better than chance. (i.e it is random)
(Rothwell & Martyn, 2000)

27. Sentiments on around classical peer review
Nicholson & Alperin, The
Winnower 2016

28. Would making reviews open change the content?
Nicholson & Alperin, The
Winnower 2016
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Yes, very different
Yes, somewhat different
Perhaps different in tone, but
not in content
No, not significantly different
it was published for others to
see
(%) of respondents

29. Would making reviews open require a lot of work?
Nicholson & Alperin, The
Winnower 2016
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
A great deal
A lot
A moderate amount
A liIle
None at all
(%) of respondents

30. “I'm an editor at two medical journals: The BMJ and BMJ Open, both of
which use open peer review. The BMJ conducted two randomised
controlled trials showing that a) signed review and b) open peer review,
with prepublication histories published next to each paper did not lower
the quality or depth of content in peer reviewers' reports. The tone was,
however, slightly more constructive”

31. “I think that if people have to sign their name to a document, they take
more care and effort to review carefully and thoughtfully.”

32. “If it had been published the remarks that are exceptionally rude and/or
stupid would probably have been avoided by the reviewers.”

33. Reasons that would incentivize scholars to
make their peer reviews publicly available.
Nicholson & Alperin, The
Winnower 2016
My performance
review/tenure
commiIee explicitly
recognize published
reviews
27%
My peers published their
reviews
26%
I was paid a small
honorarium (cash)
7%
I was paid in-kind (e.g.,
free access to journal,
waived arUcle processing
fee, etc.)
17%
The journal editor gave
me posiUve feedback on
my review
16%
Nothing could make me
do it
3%
Something else
4%

34. RECOMMEND:
MAJOR REVISIONS

36. “Scientific publishing +
Reddit = The Winnower
and hopefully a more
open world of science!!”
-Alexis Ohanian
Founder of Reddit

39. the trajectory
our authors come from all around the world
rapid growth

41. but still lots to
be done…

42. how do we incentivize
people to participate
in open post-pub
review?

43. can we automate
identification,
validation, invitation of
reviewers?

44. what else can we do to
make peer review
more robust?

45. thanks!
josh@thewinnower.com
@thewinnower
thewinnower.com