DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Survey

TL;DR

DeepSurvey-Bench is a new benchmark designed to evaluate the true academic quality of automatically generated scientific surveys across multiple dimensions, addressing limitations of existing surface-level evaluation methods.

Contribution

It introduces a comprehensive evaluation framework with annotated datasets to assess the deep academic value of generated surveys, surpassing traditional surface-level metrics.

Findings

High consistency with human judgment in academic value assessment

Addresses key limitations of existing benchmarks

Provides a multi-dimensional evaluation criteria

Abstract

The rapid development of automated scientific survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surveys.Nearly all existing evaluation benchmarks rely on flawed selection criteria such as citation counts and structural coherence to select human-written surveys as the ground truth survey datasets, and then use surface-level metrics such as structural quality and reference relevance to evaluate generated surveys.However, these benchmarks have two key issues: (1) the ground truth survey datasets are unreliable because of a lack academic dimension annotations; (2) the evaluation metrics only focus on the surface quality of the survey such as logical coherence. Both issues lead to existing benchmarks cannot assess to evaluate their deep "academic value", such as the core research objectives and the critical analysis of different studies. To address the above problems, we propose DeepSurvey-Bench, a novel benchmark designed to comprehensively evaluate the academic value of generated surveys. Specifically, our benchmark propose a comprehensive academic value evaluation criteria covering three dimensions: informational value, scholarly communication value, and research guidance value. Based on this criteria, we construct a reliable dataset with academic value annotations, and evaluate the deep academic value of the generated surveys. Extensive experimental results demonstrate that our benchmark is highly consistent with human performance in assessing the academic value of generated surveys.