When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews

TL;DR

This paper introduces a fine-grained approach to analyze conflicting peer reviews by identifying contradiction evidence and grading disagreement severity, supported by a new annotated benchmark and a multi-agent framework.

Contribution

It presents RevCI, a new annotated dataset, and IMPACT, a structured multi-agent model for detailed contradiction analysis in peer reviews, along with a compact model TIDE.

Findings

IMPACT outperforms baselines in evidence detection and disagreement grading

TIDE achieves competitive results with lower inference cost

Fine-grained analysis improves understanding of review disagreements

Abstract

Scientific peer reviews frequently contain conflicting expert judgments, and the increasing scale of conference submissions makes it challenging for Area Chairs and editors to reliably identify and interpret such disagreements. Existing approaches typically frame reviewer disagreement as binary contradiction detection over isolated sentence pairs, abstracting away the review-level context and obscuring differences in the severity of evaluative conflict. In this work, we introduce a fine-grained formulation of reviewer contradiction analysis that operates over full peer reviews by explicitly identifying contradiction evidence spans and assigning graded disagreement intensity scores. To support this task, we present RevCI, an expert-annotated benchmark of peer-review pairs with evidence-level contradiction annotations with graded intensity labels. We further propose IMPACT, a structured multi-agent framework that integrates aspect-conditioned evidence extraction, deliberative reasoning, and adjudication to model reviewer contradictions and their intensity. To support efficient deployment, we distill IMPACT into TIDE, a small language model that predicts contradiction evidence and intensity in a single forward pass. Experimental results show that IMPACT substantially outperforms strong single-agent and generic multi-agent baselines in both evidence identification and intensity agreement, while TIDE achieves competitive performance at significantly lower inference cost.