PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review

TL;DR

This paper introduces PaperAudit-Bench, a comprehensive benchmark for evaluating error detection in research papers, combining a detailed error dataset and an automated review framework to improve critical peer review using large language models.

Contribution

It presents a novel benchmark with a dataset and framework for error detection and critical review in research papers, addressing limitations of current automated peer review systems.

Findings

Error detectability varies significantly across models and detection depths.

Incorporating explicit error detection leads to stricter, more discriminative reviews.

The dataset enables training lightweight error detectors with reduced computational cost.

Abstract

Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper. In this paper, we introduce PaperAudit-Bench, which consists of two components: (1) PaperAudit-Dataset, an error dataset covering both errors identifiable within individual sections and those requiring cross-section reasoning, designed for controlled evaluation under long-context settings; and (2) PaperAudit-Review, an automated review framework that integrates structured error detection with evidence-aware review generation to support critical assessment. Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths, highlighting the difficulty of identifying such errors under long-context settings. Relative to representative automated reviewing baselines, incorporating explicit error detection into the review workflow produces systematically stricter and more discriminative evaluations, demonstrating its suitability for peer review. Finally, we show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.