Calibration with Privacy in Peer Review

TL;DR

This paper studies how to calibrate peer review scores while preserving reviewer privacy, balancing the tradeoff between protecting reviewer identities and ensuring high-quality paper acceptance.

Contribution

It provides a theoretical framework for privacy-preserving calibration, characterizes the optimal privacy-utility tradeoff, and offers efficient algorithms that achieve this optimal balance.

Findings

Established the Pareto frontier of privacy versus utility in review calibration.

Designed computationally-efficient algorithms that are Pareto optimal.

Analyzed a simplified model with two reviewers and two papers.

Abstract

Reviewers in peer review are often miscalibrated: they may be strict, lenient, extreme, moderate, etc. A number of algorithms have previously been proposed to calibrate reviews. Such attempts of calibration can however leak sensitive information about which reviewer reviewed which paper. In this paper, we identify this problem of calibration with privacy, and provide a foundational building block to address it. Specifically, we present a theoretical study of this problem under a simplified-yet-challenging model involving two reviewers, two papers, and an MAP-computing adversary. Our main results establish the Pareto frontier of the tradeoff between privacy (preventing the adversary from inferring reviewer identity) and utility (accepting better papers), and design explicit computationally-efficient algorithms that we prove are Pareto optimal.