Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards

TL;DR

This paper advocates for transforming AI conference peer review into a bi-directional system with author feedback and reviewer rewards to improve review quality and accountability amidst increasing submission volumes.

Contribution

It proposes a novel two-stage review process with author evaluations and a systematic reviewer reward system to enhance accountability and review quality in AI conferences.

Findings

Introduces a bi-directional review system with author feedback

Proposes a reviewer reward system to incentivize quality reviews

Highlights the need for community engagement in reforming peer review

Abstract

The peer review process in major artificial intelligence (AI) conferences faces unprecedented challenges with the surge of paper submissions (exceeding 10,000 submissions per venue), accompanied by growing concerns over review quality and reviewer responsibility. This position paper argues for the need to transform the traditional one-way review system into a bi-directional feedback loop where authors evaluate review quality and reviewers earn formal accreditation, creating an accountability framework that promotes a sustainable, high-quality peer review system. The current review system can be viewed as an interaction between three parties: the authors, reviewers, and system (i.e., conference), where we posit that all three parties share responsibility for the current problems. However, issues with authors can only be addressed through policy enforcement and detection tools, and ethical concerns can only be corrected through self-reflection. As such, this paper focuses on reforming reviewer accountability with systematic rewards through two key mechanisms: (1) a two-stage bi-directional review system that allows authors to evaluate reviews while minimizing retaliatory behavior, (2)a systematic reviewer reward system that incentivizes quality reviewing. We ask for the community's strong interest in these problems and the reforms that are needed to enhance the peer review process.