ScholarSearch: Benchmarking Scholar Searching Ability of LLMs

TL;DR

ScholarSearch introduces a specialized benchmark dataset designed to evaluate the complex academic search and literature retrieval capabilities of Large Language Models across multiple disciplines, emphasizing real-world research scenarios.

Contribution

It is the first dataset tailored specifically for assessing LLMs' academic search abilities, addressing limitations of existing benchmarks by focusing on practical, difficult, and broad academic information retrieval tasks.

Findings

Enhanced evaluation of LLMs in academic search tasks

Identification of current model limitations in complex literature retrieval

Benchmark promotes development of more capable academic search models

Abstract

Large Language Models (LLMs)' search capabilities have garnered significant attention. Existing benchmarks, such as OpenAI's BrowseComp, primarily focus on general search scenarios and fail to adequately address the specific demands of academic search. These demands include deeper literature tracing and organization, professional support for academic databases, the ability to navigate long-tail academic knowledge, and ensuring academic rigor. Here, we proposed ScholarSearch, the first dataset specifically designed to evaluate the complex information retrieval capabilities of Large Language Models (LLMs) in academic research. ScholarSearch possesses the following key characteristics: Academic Practicality, where question content closely mirrors real academic learning and research environments, avoiding deliberately misleading models; High Difficulty, with answers that are challenging for single models (e.g., Grok DeepSearch or Gemini Deep Research) to provide directly, often requiring at least three deep searches to derive; Concise Evaluation, where limiting conditions ensure answers are as unique as possible, accompanied by clear sources and brief solution explanations, greatly facilitating subsequent audit and verification, surpassing the current lack of analyzed search datasets both domestically and internationally; and Broad Coverage, as the dataset spans at least 15 different academic disciplines. Through ScholarSearch, we expect to more precisely measure and promote the performance improvement of LLMs in complex academic information retrieval tasks. The data is available at: https://huggingface.co/datasets/PKU-DS-LAB/ScholarSearch