DR$^{3}$-Eval: Towards Realistic and Reproducible Deep Research Evaluation

TL;DR

DR$^{3}$-Eval is a new benchmark designed to realistically and reproducibly evaluate deep research agents on complex, multimodal report generation tasks using authentic data and a comprehensive evaluation framework.

Contribution

It introduces a realistic, verifiable benchmark with a multi-dimensional evaluation framework for assessing deep research agents in complex web-like environments.

Findings

DR$^{3}$-Eval is highly challenging for current agents.

The benchmark reveals critical issues in retrieval robustness.

The evaluation correlates well with human judgments.

Abstract

Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR$^{3}$-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR$^{3}$-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR$^{3}$-Agent based on multiple state-of-the-art language models demonstrate that DR$^{3}$-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.