From Authors to Reviewers: Leveraging Rankings to Improve Peer Review

TL;DR

This paper discusses leveraging reviewer ranking data to improve peer review quality, demonstrating through simulations that reviewer rankings enhance evaluation accuracy more than author rankings alone, especially when combined.

Contribution

It introduces an approach that uses reviewer ranking information to improve paper evaluation, showing its effectiveness over author-based rankings through simulation studies.

Findings

Reviewer rankings significantly improve paper quality assessment.

Combining reviewer and author rankings yields the best evaluation accuracy.

Reviewer-based evaluation outperforms author-based methods in simulations.

Abstract

This paper is a discussion of the 2025 JASA discussion paper by Su et al. (2025). We would like to congratulate the authors on conducting a comprehensive and insightful empirical investigation of the 2023 ICML ranking data. The review quality of machine learning (ML) conferences has become a big concern in recent years, due to the rapidly growing number of submitted manuscripts. In this discussion, we propose an approach alternative to Su et al. (2025) that leverages ranking information from reviewers rather than authors. We simulate review data that closely mimics the 2023 ICML conference submissions. Our results show that (i) incorporating ranking information from reviewers can significantly improve the evaluation of each paper's quality, often outperforming the use of ranking information from authors alone; and (ii) combining ranking information from both reviewers and authors yields the most accurate evaluation of submitted papers in most scenarios.