Loss Functions, Axioms, and Peer Review

TL;DR

This paper proposes a framework inspired by empirical risk minimization to learn the community's aggregate mapping in peer review, addressing inconsistency caused by different reviewers' criteria, and identifies optimal loss hyperparameters based on axiomatic properties.

Contribution

It introduces an ERM-based approach for aggregating peer review criteria and characterizes the optimal loss hyperparameters using social choice axioms.

Findings

Identifies $L(1,1)$ loss as satisfying key axioms.

Provides a method to learn review aggregation functions.

Applied to IJCAI 2017 reviews with promising results.

Abstract

It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping. The key challenge that arises is the specification of a loss function for ERM. We consider the class of $L(p,q)$ loss functions, which is a matrix-extension of the standard class of $L_p$ losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters $p, q \in [1,\infty]$. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters $p$ and $q$. Specifically, we characterize $p=q=1$ as the only choice of these hyperparameters that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.