Least Square Calibration for Peer Review

TL;DR

This paper introduces a flexible least square calibration framework for peer review systems that improves the selection of top candidates by effectively handling various types of miscalibration and noise in reviewer ratings.

Contribution

It proposes a novel calibration method that is theoretically sound for linear scoring functions and empirically effective for broader, noisy models.

Findings

Outperforms baseline methods in synthetic experiments

Provably achieves perfect calibration under certain conditions

Effective across different types of scoring functions and noise levels

Abstract

Peer review systems such as conference paper review often suffer from the issue of miscalibration. Previous works on peer review calibration usually only use the ordinal information or assume simplistic reviewer scoring functions such as linear functions. In practice, applications like academic conferences often rely on manual methods, such as open discussions, to mitigate miscalibration. It remains an important question to develop algorithms that can handle different types of miscalibrations based on available prior knowledge. In this paper, we propose a flexible framework, namely least square calibration (LSC), for selecting top candidates from peer ratings. Our framework provably performs perfect calibration from noiseless linear scoring functions under mild assumptions, yet also provides competitive calibration results when the scoring function is from broader classes beyond linear functions and with arbitrary noise. On our synthetic dataset, we empirically demonstrate that our algorithm consistently outperforms the baseline which select top papers based on the highest average ratings.