What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review

TL;DR

This paper surveys the NLP community to identify key factors and stakeholder perspectives for improving paper-reviewer matching in peer review, emphasizing interpretability and community insights.

Contribution

It presents the first community survey on peer review matching factors, offering actionable recommendations and highlighting stakeholder perspectives.

Findings

Community values multiple factors in reviewer assignment

Stakeholders prefer transparent and interpretable matching processes

Recommendations aim to enhance conference review quality

Abstract

Both scientific progress and individual researcher careers depend on the quality of peer review, which in turn depends on paper-reviewer matching. Surprisingly, this problem has been mostly approached as an automated recommendation problem rather than as a matter where different stakeholders (area chairs, reviewers, authors) have accumulated experience worth taking into account. We present the results of the first survey of the NLP community, identifying common issues and perspectives on what factors should be considered by paper-reviewer matching systems. This study contributes actionable recommendations for improving future NLP conferences, and desiderata for interpretable peer review assignments.