Eliciting Honest Information From Authors Using Sequential Review

TL;DR

This paper introduces a sequential review mechanism that encourages authors to truthfully reveal their paper rankings, leading to higher quality accepted papers, reduced workload, and incentivizing better author behavior in peer review.

Contribution

It proposes a new sequential review mechanism that elicits truthful ranking information under realistic utility assumptions, improving conference decision quality and author incentives.

Findings

Elicits truthful ranking information from authors.

Reduces reviewing workload and increases average paper quality.

Encourages authors to produce fewer, higher-quality papers.

Abstract

In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in peer review settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) improving the quality of accepted papers, reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.