Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

TL;DR

This paper investigates how to optimally split reviewers in two-phase conference review processes and experiments, showing that random splitting performs nearly optimally in practice and providing theoretical insights into this phenomenon.

Contribution

It proves NP-hardness of the reviewer splitting problem when the set of papers needing additional review is unknown, and demonstrates the near-optimality of random reviewer splits through empirical and theoretical analysis.

Findings

Random reviewer splitting achieves near-optimal assignment similarity.

NP-hardness of the reviewer splitting problem when the additional review set is unknown.

Theoretical bounds explain when random splitting is effective.

Abstract

Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.