DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

TL;DR

DEER is a comprehensive benchmark designed to evaluate expert-level research report generation by deep research systems, addressing challenges in multifaceted quality assessment, domain-specific error detection, and evidence verification.

Contribution

The paper introduces DEER, a detailed evaluation framework with a taxonomy, rubric, and claim verification architecture for assessing deep research report quality.

Findings

Current systems produce plausible reports with external evidence

Room for improvement in fulfilling expert-level requests

DEER provides interpretable diagnostics and system insights

Abstract

Recent advances in large language models have enabled deep research systems that generate expert-level reports through multi-step reasoning and evidence-based synthesis. However, evaluating such reports remains challenging: report quality is multifaceted, making it difficult to determine what to assess and by what criteria; LLM-based judges may miss errors that require domain expertise to identify; and because deep research relies on retrieved evidence, report-wide claim verification is also necessary. To address these issues, we propose DEER, a benchmark for evaluating expert-level deep research reports. DEER systematizes evaluation criteria with an expert-developed taxonomy (7 dimensions, 25 subdimensions) operationalized as 101 fine-grained rubric items. We also provide task-specific Expert Evaluation Guidance to support LLM-based judging. Alongside rubric-based assessment, we propose a claim verification architecture that verifies both cited and uncited claims and quantifies evidence quality. Experiments show that while current deep research systems can produce structurally plausible reports that cite external evidence, there is room for improvement in fulfilling expert-level user requests and achieving logical completeness. Beyond simple performance comparisons, DEER makes system strengths and limitations interpretable and provides diagnostic signals for improvement.