Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory

TL;DR

This paper introduces a process-aware evaluation framework for Deep Research Agents, focusing on diagnosing hallucinations throughout the research process, revealing systemic weaknesses and guiding future improvements.

Contribution

It proposes the PIES Taxonomy and a fine-grained evaluation framework to analyze hallucinations in DRAs along functional and error dimensions.

Findings

No current DRA achieves robust reliability.

Hallucination propagation and biases are key failure sources.

DeepHalluBench effectively isolates hallucination-prone tasks.

Abstract

Diagnosing the failure mechanisms of Deep Research Agents (DRAs) remains a critical challenge. Existing benchmarks predominantly rely on end-to-end evaluation, obscuring critical intermediate hallucinations, such as flawed planning, that accumulate throughout the research trajectory. To bridge this gap, we propose a shift from outcome-based to process-aware evaluation by auditing the full research trajectory. We introduce the PIES Taxonomy to categorize hallucinations along functional components (Planning vs. Summarization) and error properties (Explicit vs. Implicit). We instantiate this taxonomy into a fine-grained evaluation framework that decomposes the trajectory to rigorously quantify these hallucinations. Leveraging this framework to isolate 100 distinctively hallucination-prone tasks including adversarial scenarios, we curate DeepHalluBench. Experiments on six state-of-theart DRAs reveal that no system achieves robust reliability. Furthermore, our diagnostic analysis traces the etiology of these failures to systemic deficits, specifically hallucination propagation and cognitive biases, providing foundational insights to guide future architectural optimization. Data and code are available at https://github.com/yuhao-zhan/DeepHalluBench.