ReportLogic: Evaluating Logical Quality in Deep Research Reports

TL;DR

ReportLogic introduces a comprehensive benchmark and evaluation framework for assessing the logical quality of deep research reports generated by LLMs, emphasizing traceability, understanding, and verification of claims.

Contribution

It presents a hierarchical taxonomy for logical evaluation, a human-annotated dataset, and an open-source LogicJudge model for scalable, robust assessment of report quality.

Findings

LLM judges are often misled by superficial cues like verbosity

Adversarial attacks reveal vulnerabilities in current evaluation methods

LogicJudge can guide improvements in report logical reliability

Abstract

Users increasingly rely on Large Language Models (LLMs) for Deep Research, using them to synthesize diverse sources into structured reports that support understanding and action. In this context, the practical reliability of such reports hinges on logical quality: whether the report's claims and arguments are explicitly supported and can be trusted as a basis for downstream use, rather than merely appearing fluent or informative. However, current evaluation frameworks largely overlook this requirement. To bridge this gap, we introduce ReportLogic, a benchmark that quantifies report-level logical quality through a reader-centric lens of auditability. Specifically, ReportLogic adopts a hierarchical taxonomy that evaluates whether readers can (1) trace an on-topic report structure with a unified analytical arc (Macro-Logic), (2) understand the progression with necessary context (Expositional-Logic), and (3) verify conclusions via explicit claim--support (Structural-Logic). Based on this taxonomy, we construct a human-annotated rubric-guided dataset and train an open-source LogicJudge for scalable evaluation. We further evaluate judge robustness via adversarial attacks, showing that off-the-shelf LLM judges are frequently influenced by superficial cues (e.g., verbosity), and reasoning modes can mask broken support relations. Overall, our results provide actionable guidance for building more robust logic evaluators and improving the logical reliability of LLM-generated reports.