Peer Prediction for Peer Review: Designing a Marketplace for Ideas

TL;DR

This paper proposes a novel platform for academic peer review that leverages peer prediction algorithms to improve review accuracy and timeliness, especially for early-stage research, addressing key inefficiencies in scientific publishing.

Contribution

It introduces a peer review marketplace utilizing a peer prediction algorithm based on Peer Truth Serum, integrating human raters and machine learning to enhance review quality and transparency.

Findings

Improved accuracy in early research peer review.

Enhanced incentives for sharing negative results.

Potential reduction in publication bias.

Abstract

The paper describes a potential platform to facilitate academic peer review with emphasis on early-stage research. This platform aims to make peer review more accurate and timely by rewarding reviewers on the basis of peer prediction algorithms. The algorithm uses a variation of Peer Truth Serum for Crowdsourcing (Radanovic et al., 2016) with human raters competing against a machine learning benchmark. We explain how our approach addresses two large productive inefficiencies in science: mismatch between research questions and publication bias. Better peer review for early research creates additional incentives for sharing it, which simplifies matching ideas to teams and makes negative results and p-hacking more visible.