How anonymous post-publication peer review uncovers bad science
1. How anonymous
post-publication peer review
uncovers bad science
Leonid Schneider,
science journalist
with Laborjournal
leonid.schneider@gmail.com
Twitter: @schneiderleonid
2. Junior scientists are often told by their advisors:
- If you can deliver this result,
you will publish a nice paper and have a job
- If you don’t deliver this result,
you will not publish any paper and have no job
Is bad science
individual or systemic failure?
3. $$$
Authors and institutions have little incentive to
produce reliable quality science
Paper-to-funding convertion
Funding used for research…
4. Journals and funding agencies prefer
simplistic, but sensationalist “break-through” science
• Cancer cure!
• Stem cells /”Reprogramming”
• One-Gene-Phenotype models
• Translational/Commercial
potential
Biological systems
are very complicated,
but in biological papers
simplicity rules!
5. Scientists occasionally help data to fit their
theoretical model for a publication
• Selective data acquisition and evaluation
(very common)
• “Adjustments” or manipulation of data
(less common)
• Data falsification / fraud
(very rare)
6. Peer review weeds out bad science. Really?
• Data is submitted on trust as
being honest/reliable
• Peer Reviewers are scientist
colleagues, not data specialists
• Peer Reviewers only analyse
science, not data integrity
• Peer review is not always done
diligently enough
How did this pass
peer review????
8. - Journal Editors
- Decide on Quality,
Novelty, Impact
- Appoint peer
reviewers
- Make final decisions
- Peer Reviewers
- 1-4 people
- Unknown to authors
or readers
- Potential COI,
personal animosities,
lack of competence…
$$$
Too many financial and personal interests involved
Years and years of research…
10. A peer-reviewed paper is a badge of honour
Things surely changed for him
since he published in Nature…
• Publications are public
evidence of success
• They are to be admired
and not questioned
• Often not the content
counts, but where it is
published
11. Scientists waste time, money and their careers trying to
reproduce unreliable or manipulated results
• Poor reproducibility in combination with high competition
undermines productivity, but also work moral, trust and
motivation
• It leads to even more data manipulation and fraud in science
12. What do you do if you spot data irregularities or
irreproducibility in a published paper?
1. Write to authors
2. Write to journal
3. Write to authors’
institution
13. What happens if a published paper is reported
to be wrong or even contain manipulated data?
1. Correction (rare)
2. Retraction (even rarer)
3. Nothing (most common)
14. Your paper is wrong,
professor!
See you at the
exam…
Individual criticisms are unwelcome and dangerous
• Financial interests behind publications
prevent institutional investigations
• Institutions often refuse to react to
anonymous hints
• Whistle-blowers are often punished or
dismissed as incompetent or malicious
15. Solution: make valid criticisms public,
but anonymously!
• Publicly available valid criticisms
are much more difficult to be
ignored
• Whistle-blowers are protected by
the anonymity under which they
are free to post concerns
19. Or, you can post evidence of data irregularities on PubPeer,
also anonymously
20. I have some issues
with your paper, Sir!
Pros and cons of anonymous commenting
(aka witch-hunts)
• Protects whistle-blowers
• Only objective evidence and
arguments matter, not who has
raised them or why or where
• Unsubstantiated claims,
personal insults
• Sock-puppeting (also by authors!)
Against:
For:
22. Blog post at Laborjournal about this interesting and
useful experience:
http://www.laborjournal.de/blog/?p=8281
Autors also reply to
PubPeer criticisms, often
constructively
23. STAP: one of the biggest fraud scandals uncovered,
thanks to post-publication peer review on PubPeer
24. Photo credit: Maigrot/REA
The case Olivier Voinnet
• PhD with Sir David
Baulcombe at
The Sainsbury
Laboratory, Norwich
• Research leader at
CNRS institute in
Strasbourg (age 33)
• Professor at ETH
Zürich (since 2010)
25. Photo credit: Maigrot/REA
The case Olivier Voinnet
• EMBO Gold Medal
(2009)
• EMBO Member
• EMBO Young
Investigator grant
• ERC start-up grant
• Max-Rössler-Prize
(ETH Zürich, 2013)
26. It started with people finding irregularities in
David Baulcombe’s papers