Recommending Best Paper Awards for ML/AI Conferences via the Isotonic Mechanism

TL;DR

This paper proposes an author-assisted isotonic mechanism to improve the fairness and accuracy of best paper award selection in ML/AI conferences by incentivizing truthful reporting and adjusting review scores.

Contribution

It introduces a novel isotonic mechanism for eliciting author rankings, proves truthfulness under realistic conditions, and extends it to handle overlapping authorship scenarios.

Findings

Authors are incentivized to report truthfully under the proposed mechanism.

The mechanism improves the quality of selected award papers in simulations.

Validation of convex utility assumption using real conference review data.

Abstract

Machine learning and artificial intelligence conferences such as NeurIPS and ICML now regularly receive tens of thousands of submissions, posing significant challenges to maintaining the quality and consistency of the peer review process. This challenge is particularly acute for best paper awards, which are an important part of the peer review process, yet whose selection has increasingly become a subject of debate in recent years. In this paper, we introduce an author-assisted mechanism to facilitate the selection of best paper awards. Our method employs the Isotonic Mechanism for eliciting authors' assessments of their own submissions in the form of a ranking, which is subsequently utilized to adjust the raw review scores for optimal estimation of the submissions' ground-truth quality. We demonstrate that authors are incentivized to report truthfully when their utility is a convex additive function of the adjusted scores, and we validate this convexity assumption for best paper awards using publicly accessible review data of ICLR from 2019 to 2023 and NeurIPS from 2021 to 2023. Crucially, in the special case where an author has a single quota -- that is, may nominate only one paper -- we prove that truthfulness holds even when the utility function is merely nondecreasing and additive. This finding represents a substantial relaxation of the assumptions required in prior work. For practical implementation, we extend our mechanism to accommodate the common scenario of overlapping authorship. Finally, simulation results demonstrate that our mechanism significantly improves the quality of papers selected for awards.