SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation

TL;DR

SurGE introduces a comprehensive benchmark and evaluation framework for scientific survey generation, addressing the lack of standardized tools and revealing current limitations of large language models in this task.

Contribution

It provides a new benchmark dataset, an automated multi-dimensional evaluation framework, and open-sources code and data to advance research in automated scientific survey generation.

Findings

Large language models still struggle with survey generation complexity.

Significant performance gap exists among current LLM-based methods.

The benchmark reveals areas for future improvement in survey generation.

Abstract

The rapid growth of academic literature makes the manual creation of scientific surveys increasingly infeasible. While large language models show promise for automating this process, progress in this area is hindered by the absence of standardized benchmarks and evaluation protocols. To bridge this critical gap, we introduce SurGE (Survey Generation Evaluation), a new benchmark for scientific survey generation in computer science. SurGE consists of (1) a collection of test instances, each including a topic description, an expert-written survey, and its full set of cited references, and (2) a large-scale academic corpus of over one million papers. In addition, we propose an automated evaluation framework that measures the quality of generated surveys across four dimensions: comprehensiveness, citation accuracy, structural organization, and content quality. Our evaluation of diverse LLM-based methods demonstrates a significant performance gap, revealing that even advanced agentic frameworks struggle with the complexities of survey generation and highlighting the need for future research in this area. We have open-sourced all the code, data, and models at: https://github.com/oneal2000/SurGE