References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation

TL;DR

This paper investigates how the choice of reference summaries affects the reliability of evaluation metrics in summarization, revealing significant instability and weak correlation with human judgments, especially for large language models.

Contribution

It systematically analyzes the impact of reference set variation on popular metrics across multiple datasets, highlighting the need to consider reference diversity for more reliable evaluation.

Findings

Metrics show significant instability depending on reference sets.

ROUGE scores vary with different reference summaries, affecting model rankings.

Weak correlation found between automatic metrics and human judgments for LLM outputs.

Abstract

Human language production exhibits remarkable richness and variation, reflecting diverse communication styles and intents. However, this variation is often overlooked in summarization evaluation. While having multiple reference summaries is known to improve correlation with human judgments, the impact of the reference set on reference-based metrics has not been systematically investigated. This work examines the sensitivity of widely used reference-based metrics in relation to the choice of reference sets, analyzing three diverse multi-reference summarization datasets: SummEval, GUMSum, and DUC2004. We demonstrate that many popular metrics exhibit significant instability. This instability is particularly concerning for n-gram-based metrics like ROUGE, where model rankings vary depending on the reference sets, undermining the reliability of model comparisons. We also collect human judgments on LLM outputs for genre-diverse data and examine their correlation with metrics to supplement existing findings beyond newswire summaries, finding weak-to-no correlation. Taken together, we recommend incorporating reference set variation into summarization evaluation to enhance consistency alongside correlation with human judgments, especially when evaluating LLMs.