Recommended
More Related Content
Viewers also liked
Viewers also liked (19)
Similar to Peer Review in the LiquidPub project
Similar to Peer Review in the LiquidPub project (20)
More from Aliaksandr Birukou
More from Aliaksandr Birukou (19)
Recently uploaded
Recently uploaded (20)
Peer Review in the LiquidPub project
- 2. Metric Dimensions Quality Fairness Efficiency Statistics Kendall Distance Divergence Disagreement Biases Robustness Unbiasing Effort invariant alternatives Effort vs. quality Min. criteria
- 4. Divergence Metric ©√ Nt = 1/n Normalized t Nt = n/n N-Divergence Ndiv(1,n, C) = n-1/n ©√ When the second ranking is random, we have: indipendent correlated inv. correlated
- 5. prior vs. after discussion ca. 74 (36%) contributions have been effected by the discussion phase
- 6. Results: peer review ranking vs. citation count Div Normalized t
- 9. Computed Normalized Rating Biases C1 C2 C3 C4 top accepting 2,66 3,44 1,52 1,17 top rejecting -1,74 -2,78 -2,06 -1,17 > + |min bias| 13% 5% 9% 7% < - |min bias| 12% 4% 8% 7% C1 C2 C3 C4 Unbiasing effect (divergence) 13% 9% 11% 14% Unbiasing effect (reviewers affected) 10 16 5 4
- 11. Disagreement vs number of reviews
- 12. The road ahead Real-Time accuracy estimation Speed Ranking Ranking vs. marking
- 13. Thank you
Editor's Notes
- For instance if t =1, n = 4 p1(0) = probability that the first number is different from the first in the ordered ranking p1(1) 0 probability that the first number is equal to the first in the ordered ranking
- Check numbers
- Notice that if we restricted the world to the top N papers and accepted only 30% of them, or 20% of them, we would have essentially a random process. The bottom right suggests that it is not true that for weaker papers we can have a clear decision We were not looking for this results Acceptance rate is over-rated
- Randomness in review Due to many reasons: subjective reviews limited size of the sample for a given reviewer absolute subjective marking vs. relative ranking