Inducing Honest Reporting Without Observing Outcomes: An Application to the Peer-Review Process

TL;DR

This paper introduces a new scoring method based on proper scoring rules that incentivizes honest expert reporting without needing observable outcomes, demonstrated through an application to peer review.

Contribution

It develops a pairwise comparison scoring approach that promotes truthful reporting in settings lacking observable outcomes, applied specifically to peer review.

Findings

Encourages honest reviews in peer review process

Produces more accurate reviews than traditional methods

Works under Bayesian decision-making assumptions

Abstract

When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In this paper, we propose a scoring method built on strictly proper scoring rules to induce honest reporting without assuming observable outcomes. Our method provides scores based on pairwise comparisons between the reports made by each pair of experts in the group. For ease of exposition, we introduce our scoring method by illustrating its application to the peer-review process. In order to do so, we start by modeling the peer-review process using a Bayesian model where the uncertainty regarding the quality of the manuscript is taken into account. Thereafter, we introduce our scoring method to evaluate the reported reviews. Under the assumptions that reviewers are Bayesian decision-makers and that they cannot influence the reviews of other reviewers, we show that risk-neutral reviewers strictly maximize their expected scores by honestly disclosing their reviews. We also show how the group's scores can be used to find a consensual review. Experimental results show that encouraging honest reporting through the proposed scoring method creates more accurate reviews than the traditional peer-review process.