Beyond Retrieval: A Modular Benchmark for Academic Deep Research Agents

TL;DR

This paper introduces ADRA-Bank, a modular benchmark and evaluation paradigm for academic deep research agents, addressing gaps in existing benchmarks by focusing on high-level planning, reasoning, and academic domain specificity.

Contribution

It presents a new human-annotated dataset and a modular evaluation framework tailored for academic research agents, emphasizing planning, retrieval, and reasoning capabilities.

Findings

Agents excel in specialized tasks but struggle with multi-source retrieval.

High-level planning is key to unlocking reasoning in foundational LLMs.

The benchmark exposes actionable failure modes for future improvements.

Abstract

A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, existing benchmarks often focus narrowly on retrieval while neglecting high-level planning and reasoning. Second, existing benchmarks favor general domains over the academic domains that are the core application for DR agents. To address these gaps, we introduce ADRA-Bank, a modular benchmark for Academic DR Agents. Grounded in academic literature, our benchmark is a human-annotated dataset of 200 instances across 10 academic domains, including both research and review papers. Furthermore, we propose a modular Evaluation Paradigm for Academic DR Agents (ADRA-Eval), which leverages the rich structure of academic papers to assess the core capabilities of planning, retrieval, and reasoning. It employs two complementary modes: an end-to-end evaluation for \task agents and an isolated evaluation for foundational LLMs as potential backbones. Results reveal uneven capabilities: while agents show specialized strengths, they struggle with multi-source retrieval and cross-field consistency. Moreover, improving high-level planning capability is the crucial factor for unlocking the reasoning potential of foundational LLMs as backbones. By exposing these actionable failure modes, ADRA-Bank provides a diagnostic tool to guide the development of more reliable automatic academic research assistants.