IDRBench: Interactive Deep Research Benchmark
Yingchaojie Feng, Qiang Huang, Xiaoya Xie, Zhaorui Yang, Jun Yu, Wei Chen, Anthony K. H. Tung

TL;DR
IDRBench is a novel benchmark designed to evaluate interactive deep research agents powered by LLMs, emphasizing the importance of dynamic user interaction, measuring both benefits and costs of interaction.
Contribution
It introduces the first systematic benchmark for interactive deep research, incorporating a modular framework, user simulation, and interaction-aware evaluation metrics.
Findings
Interaction improves research quality and robustness across models.
Trade-offs exist between interaction benefits and efficiency.
Interaction effects often surpass differences in model capacity.
Abstract
Deep research agents powered by Large Language Models (LLMs) can perform multi-step reasoning, web exploration, and long-form report generation. However, most existing systems operate in an autonomous manner, assuming fully specified user intent and evaluating only final outputs. In practice, research goals are often underspecified and evolve during exploration, making sustained interaction essential for robust alignment. Despite its importance, interaction remains largely invisible to existing deep research benchmarks, which neither model dynamic user feedback nor quantify its costs. We introduce IDRBench, the first benchmark for systematically evaluating interactive deep research. IDRBench combines a modular multi-agent research framework with on-demand interaction, a scalable reference-grounded user simulator, and an interaction-aware evaluation suite that jointly measures…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Artificial Intelligence in Healthcare and Education · Multimodal Machine Learning Applications
