From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review

TL;DR

This paper proposes a novel peer review mechanism using LLMs for pairwise manuscript comparisons, which improves quality assessment but introduces biases affecting diversity and equity.

Contribution

It introduces a new LLM-based pairwise comparison approach for peer review, moving beyond traditional scoring methods and highlighting both benefits and biases.

Findings

Outperforms traditional rating methods in identifying high-impact papers.

Reveals biases towards less novel topics and institutional imbalance.

Demonstrates potential and challenges of LLMs in peer review.

Abstract

The advent of large language models (LLMs) offers unprecedented opportunities to reimagine peer review beyond the constraints of traditional workflows. Despite these opportunities, prior efforts have largely focused on replicating traditional review workflows with LLMs serving as direct substitutes for human reviewers, while limited attention has been given to exploring new paradigms that fundamentally rethink how LLMs can participate in the academic review process. In this paper, we introduce and explore a novel mechanism that employs LLM agents to perform pairwise comparisons among manuscripts instead of individual scoring. By aggregating outcomes from substantial pairwise evaluations, this approach enables a more accurate and robust measure of relative manuscript quality. Our experiments demonstrate that this comparative approach significantly outperforms traditional rating-based methods in identifying high-impact papers. However, our analysis also reveals emergent biases in the selection process, notably a reduced novelty in research topics and an increased institutional imbalance. These findings highlight both the transformative potential of rethinking peer review with LLMs and critical challenges that future systems must address to ensure equity and diversity.