SurveyEval: Towards Comprehensive Evaluation of LLM-Generated Academic Surveys

TL;DR

SurveyEval is a comprehensive benchmark designed to evaluate the quality, coherence, and accuracy of LLM-generated academic surveys, addressing the challenge of assessing complex AI-generated content.

Contribution

The paper introduces SurveyEval, a novel, scalable benchmark for evaluating LLM-generated surveys across multiple dimensions and subjects, enhancing evaluation methods.

Findings

Specialized survey-generation systems outperform general models.

Evaluation framework aligns well with human judgment.

SurveyEval covers diverse subjects and criteria.

Abstract

LLM-based automatic survey systems are transforming how users acquire information from the web by integrating retrieval, organization, and content synthesis into end-to-end generation pipelines. While recent works focus on developing new generation pipelines, how to evaluate such complex systems remains a significant challenge. To this end, we introduce SurveyEval, a comprehensive benchmark that evaluates automatically generated surveys across three dimensions: overall quality, outline coherence, and reference accuracy. We extend the evaluation across 7 subjects and augment the LLM-as-a-Judge framework with human references to strengthen evaluation-human alignment. Evaluation results show that while general long-text or paper-writing systems tend to produce lower-quality surveys, specialized survey-generation systems are able to deliver substantially higher-quality results. We envision SurveyEval as a scalable testbed to understand and improve automatic survey systems across diverse subjects and evaluation criteria.