Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback

TL;DR

This paper advocates for using LLMs as mentoring and feedback tools to improve reviewer quality and address the Reviewer Gap, promoting a sustainable peer review process in AI research.

Contribution

It introduces a human-centered paradigm shift, proposing LLM-assisted mentoring and feedback systems to enhance reviewer skills and review quality.

Findings

Proposes LLM-assisted mentoring to develop reviewer expertise

Introduces feedback systems to improve review quality

Supports sustainable peer review ecosystem

Abstract

The rapid expansion of AI research has intensified the Reviewer Gap, threatening the peer-review sustainability and perpetuating a cycle of low-quality evaluations. This position paper critiques existing LLM approaches that automatically generate reviews and argues for a paradigm shift that positions LLMs as tools for assisting and educating human reviewers. We define the core principles of high-quality peer review and propose two complementary systems grounded in these foundations: (i) an LLM-assisted mentoring system that cultivates reviewers' long-term competencies, and (ii) an LLM-assisted feedback system that helps reviewers refine the quality of their reviews. This human-centered approach aims to strengthen reviewer expertise and contribute to building a more sustainable scholarly ecosystem.