Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
Andrii Zahorodnii, Jasper J.F. van den Bosch, Ian Charest, Christopher Summerfield, Ila R. Fiete

TL;DR
This paper explores an open, bottom-up peer review system using community consensus and Bayesian methods, demonstrating improved accuracy and robustness in assessing scientific paper quality compared to traditional closed review.
Contribution
It introduces a novel open peer review framework that leverages reviewer consensus, Bayesian estimation, and reviewer scoring to enhance fairness, reliability, and scalability of scientific evaluation.
Findings
Reviewer agreement varies significantly across reviewers.
Bayesian methods improve paper quality assessment accuracy.
User-generated reviewer ratings can compensate for unreliable reviewers.
Abstract
Traditional closed peer review systems, which have played a central role in scientific publishing, are often slow, costly, non-transparent, stochastic, and possibly subject to biases - factors that can impede scientific progress and undermine public trust. Here, we propose and examine the efficacy and accuracy of an alternative form of scientific peer review: through an open, bottom-up process. First, using data from two major scientific conferences (CCN2023 and ICLR2023), we highlight how high variability of review scores and low correlation across reviewers presents a challenge for collective review. We quantify reviewer agreement with community consensus scores and use this as a reviewer quality estimator, showing that surprisingly, reviewer quality scores are not correlated with authorship quality. Instead, we reveal an inverted U-shape relationship, where authors with intermediate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Text Analysis Techniques · Educational Technology and Assessment · Expert finding and Q&A systems
