Re-evaluating Evaluation in Text Summarization

TL;DR

This paper critically re-examines the reliability of standard automatic evaluation metrics in text summarization, revealing that conclusions from older datasets may not apply to modern systems and datasets.

Contribution

It provides a comprehensive re-evaluation of evaluation metrics using recent datasets and system outputs, highlighting the need to reconsider current evaluation practices.

Findings

Evaluation metrics' reliability varies across datasets and systems

Older dataset conclusions do not always hold for modern systems

Current metrics may need revision for better assessment

Abstract

Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems.