Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

TL;DR

This paper introduces Beyond Rating, a comprehensive evaluation framework for AI peer review that emphasizes textual justification over scalar scores, and proposes new metrics and datasets to improve alignment with human judgment.

Contribution

It presents a holistic evaluation framework and novel metrics for AI reviews, emphasizing textual content and argumentative quality, moving beyond traditional rating prediction benchmarks.

Findings

Text-centric metrics, especially recall of weakness arguments, correlate strongly with human preferences.

Traditional n-gram metrics fail to reflect human evaluation of reviews.

Aligning AI critique focus with human experts improves automated scoring reliability.

Abstract

The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks that treat reviewing primarily as a rating prediction task. We argue that the utility of a review lies in its textual justification--its arguments, questions, and critique--rather than a scalar score. To address this, we introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions: Content Faithfulness, Argumentative Alignment, Focus Consistency, Question Constructiveness, and AI-Likelihood. Notably, we propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews, rigorously filtered to remove procedural noise. Extensive experiments demonstrate that while traditional n-gram metrics fail to reflect human preferences, our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy. These findings establish that aligning AI critique focus with human experts is a prerequisite for reliable automated scoring, offering a robust standard for future research.