Reviewing the Reviewer: Elevating Peer Review Quality through LLM-Guided Feedback

TL;DR

This paper presents an LLM-based framework for enhancing peer review quality by decomposing reviews, detecting multiple issues, and generating targeted, guideline-aware feedback, significantly improving review quality.

Contribution

It introduces a novel neurosymbolic detection method and template refinement process for multi-issue review analysis, advancing automated peer review support.

Findings

Outperforms zero-shot LLM baselines in detection accuracy

Improves review quality by up to 92.4%

Provides LazyReviewPlus dataset with 1,309 labeled sentences

Abstract

Peer review is central to scientific quality, yet reliance on simple heuristics -- lazy thinking -- has lowered standards. Prior work treats lazy thinking detection as a single-label task, but review segments may exhibit multiple issues, including broader clarity problems, or specificity issues. Turning detection into actionable improvements requires guideline-aware feedback, which is currently missing. We introduce an LLM-driven framework that decomposes reviews into argumentative segments, identifies issues via a neurosymbolic module combining LLM features with traditional classifiers, and generates targeted feedback using issue-specific templates refined by a genetic algorithm. Experiments show our method outperforms zero-shot LLM baselines and improves review quality by up to 92.4\%. We also release LazyReviewPlus, a dataset of 1,309 sentences labeled for lazy thinking and specificity.