Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

TL;DR

This study introduces a new framework for measuring review quality in AI conferences, revealing no consistent decline over time and offering insights into review practices and standards.

Contribution

It presents a novel multi-dimensional schema and standardized approach for assessing review quality, enabling cross-temporal and cross-venue comparisons.

Findings

No consistent decline in review quality over time

Diversity in review formats across venues

Recommendations for future empirical review studies

Abstract

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.