ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agents

TL;DR

ResearchArena is a benchmark designed to evaluate large language models' ability to conduct academic surveys by simulating the research process in information discovery, selection, and organization, highlighting current limitations and future opportunities.

Contribution

This paper introduces ResearchArena, a novel benchmark for assessing LLMs' research capabilities, including a comprehensive offline environment and evaluation framework for academic survey tasks.

Findings

LLMs underperform compared to keyword-based retrieval methods

Recent reasoning models like DeepSeek-R1 show improved zero-shot performance

Significant opportunities exist for advancing LLMs in autonomous research

Abstract

Large language models (LLMs) excel across many natural language processing tasks but face challenges in domain-specific, analytical tasks such as conducting research surveys. This study introduces ResearchArena, a benchmark designed to evaluate LLMs' capabilities in conducting academic surveys -- a foundational step in academic research. ResearchArena models the process in three stages: (1) information discovery, identifying relevant literature; (2) information selection, evaluating papers' relevance and impact; and (3) information organization, structuring knowledge into hierarchical frameworks such as mind-maps. Notably, mind-map construction is treated as a bonus task, reflecting its supplementary role in survey-writing. To support these evaluations, we construct an offline environment of 12M full-text academic papers and 7.9K survey papers. To ensure ethical compliance, we do not redistribute copyrighted materials; instead, we provide code to construct the environment from the Semantic Scholar Open Research Corpus (S2ORC). Preliminary evaluations reveal that LLM-based approaches underperform compared to simpler keyword-based retrieval methods, though recent reasoning models such as DeepSeek-R1 show slightly better zero-shot performance. These results underscore significant opportunities for advancing LLMs in autonomous research. We open-source the code to construct the ResearchArena benchmark at https://github.com/cxcscmu/ResearchArena.