The AI Imperative: Scaling High-Quality Peer Review in Machine Learning

TL;DR

This paper emphasizes the urgent need for AI-assisted peer review in machine learning to address the scalability crisis, proposing a collaborative ecosystem leveraging Large Language Models to improve review quality and efficiency.

Contribution

It advocates for developing AI tools to augment human peer review processes, outlining roles, research agenda, and ethical considerations for future systems.

Findings

AI can assist in factual verification of papers

Structured review data is crucial for AI system development

AI-augmented review can improve scalability and quality

Abstract

Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite capacity of qualified reviewers, leading to concerns about review quality, consistency, and reviewer fatigue. This position paper argues that AI-assisted peer review must become an urgent research and infrastructure priority. We advocate for a comprehensive AI-augmented ecosystem, leveraging Large Language Models (LLMs) not as replacements for human judgment, but as sophisticated collaborators for authors, reviewers, and Area Chairs (ACs). We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making. Crucially, we contend that the development of such systems hinges on access to more granular, structured, and ethically-sourced peer review process data. We outline a research agenda, including illustrative experiments, to develop and validate these AI assistants, and discuss significant technical and ethical challenges. We call upon the ML community to proactively build this AI-assisted future, ensuring the continued integrity and scalability of scientific validation, while maintaining high standards of peer review.