SGSimEval: A Comprehensive Multifaceted and Similarity-Enhanced Benchmark for Automatic Survey Generation Systems

TL;DR

SGSimEval is a new comprehensive benchmark for automatic survey generation systems that combines multiple evaluation metrics, including human preferences, to better assess system quality across outline, content, and references.

Contribution

It introduces a multifaceted evaluation framework that integrates LLM-based scoring, quantitative metrics, and human preferences for assessing survey generation systems.

Findings

Current systems excel in outline generation but need improvement in content and references.

Evaluation metrics correlate well with human assessments.

SGSimEval provides a more balanced and human-aligned evaluation approach.

Abstract

The growing interest in automatic survey generation (ASG), a task that traditionally required considerable time and effort, has been spurred by recent advances in large language models (LLMs). With advancements in retrieval-augmented generation (RAG) and the rising popularity of multi-agent systems (MASs), synthesizing academic surveys using LLMs has become a viable approach, thereby elevating the need for robust evaluation methods in this domain. However, existing evaluation methods suffer from several limitations, including biased metrics, a lack of human preference, and an over-reliance on LLMs-as-judges. To address these challenges, we propose SGSimEval, a comprehensive benchmark for Survey Generation with Similarity-Enhanced Evaluation that evaluates automatic survey generation systems by integrating assessments of the outline, content, and references, and also combines LLM-based scoring with quantitative metrics to provide a multifaceted evaluation framework. In SGSimEval, we also introduce human preference metrics that emphasize both inherent quality and similarity to humans. Extensive experiments reveal that current ASG systems demonstrate human-comparable superiority in outline generation, while showing significant room for improvement in content and reference generation, and our evaluation metrics maintain strong consistency with human assessments.