SciNetBench: A Relation-Aware Benchmark for Scientific Literature Retrieval Agents

TL;DR

SciNetBench is a new benchmark designed to evaluate scientific literature retrieval agents' ability to understand and utilize relations within scientific papers, addressing a key limitation of current content-focused retrieval methods.

Contribution

We introduce SciNetBench, the first relation-aware benchmark for scientific literature retrieval, and demonstrate the significant performance gap in current agents' ability to handle relational information.

Findings

Current retrieval agents perform below 20% accuracy on relation-aware tasks.

Providing relational ground truth improves review quality by 23.4%.

Relational understanding is crucial for effective scientific literature retrieval.

Abstract

The rapid development of AI agent has spurred the development of advanced research tools, such as Deep Research. Achieving this require a nuanced understanding of the relations within scientific literature, surpasses the scope of keyword-based or embedding-based retrieval. Existing retrieval agents mainly focus on the content-level similarities and are unable to decode critical relational dynamics, such as identifying corroborating or conflicting studies or tracing technological lineages, all of which are essential for a comprehensive literature review. Consequently, this fundamental limitation often results in a fragmented knowledge structure, misleading sentiment interpretation, and inadequate modeling of collective scientific progress. To investigate relation-aware retrieval more deeply, we propose SciNetBench, the first Scientific Network Relation-aware Benchmark for literature retrieval agents. Constructed from a corpus of over 18 million AI papers, our benchmark systematically evaluates three levels of relations: ego-centric retrieval of papers with novel knowledge structures, pair-wise identification of scholarly relationships, and path-wise reconstruction of scientific evolutionary trajectories. Through extensive evaluation of three categories of retrieval agents, we find that their accuracy on relation-aware retrieval tasks often falls below 20%, revealing a core shortcoming of current retrieval paradigms. Notably, further experiments on the literature review tasks demonstrate that providing agents with relational ground truth leads to a substantial 23.4% performance improvement in the review quality, validating the critical importance of relation-aware retrieval. We publicly release our benchmark at https://anonymous.4open.science/r/SciNetBench/ to support future research on advanced retrieval systems.