SurveyLens: A Research Discipline-Aware Benchmark for Automatic Survey Generation

TL;DR

SurveyLens introduces a discipline-aware benchmark and evaluation framework for automatic survey generation, addressing biases in existing metrics and aiding researchers across diverse academic fields.

Contribution

It presents the first discipline-specific benchmark, SurveyLens-1k, and a dual evaluation framework combining rubric and canonical alignment assessments.

Findings

Different ASG methods show varied strengths across disciplines.

Discipline-aware evaluation reveals gaps in current ASG systems.

SurveyLens guides better tool selection for high-quality, discipline-specific surveys.

Abstract

The exponential growth of scientific literature has driven the evolution of Automatic Survey Generation (ASG) from simple pipelines to multi-agent frameworks and commercial Deep Research agents. However, current ASG evaluation methods rely on generic metrics and are heavily biased toward Computer Science (CS), failing to assess whether ASG methods adhere to the distinct standards of various academic disciplines. Consequently, researchers, especially those outside CS, lack clear guidance on using ASG systems to yield high-quality surveys compliant with specific discipline standards. To bridge this gap, we introduce SurveyLens, the first discipline-aware benchmark evaluating ASG methods across diverse research disciplines. We construct SurveyLens-1k, a curated dataset of 1,000 high-quality human-written surveys spanning 10 disciplines. Subsequently, we propose a dual-lens evaluation framework: (1) Discipline-Aware Rubric Evaluation, which utilizes LLMs with human preference-aligned weights to assess adherence to domain-specific writing standards; and (2) Canonical Alignment Evaluation to rigorously measure content coverage and synthesis quality against human-written survey papers. We conduct extensive experiments by evaluating 11 state-of-the-art ASG methods on SurveyLens, including Vanilla LLMs, ASG systems, and Deep Research agents. Our analysis reveals the distinct strengths and weaknesses of each paradigm across fields, providing essential guidance for selecting tools tailored to specific disciplinary requirements.