Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries

TL;DR

This paper critically analyzes existing summarization evaluation metrics like ROUGE and BERTScore, revealing they measure topic similarity rather than true information overlap, and proposes a new method that directly assesses information quality.

Contribution

It demonstrates that current metrics do not accurately measure information overlap and introduces a simple, interpretable evaluation method that better aligns with information quality.

Findings

Current metrics mainly measure topic similarity, not information overlap.

Existing metrics do not reliably reflect the true information quality of summaries.

Proposed method provides clearer insights into model behavior and summary quality.

Abstract

Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary's information quality by calculating how much information the summaries have in common. In this work, we analyze the token alignments used by ROUGE and BERTScore to compare summaries and argue that their scores largely cannot be interpreted as measuring information overlap, but rather the extent to which they discuss the same topics. Further, we provide evidence that this result holds true for many other summarization evaluation metrics. The consequence of this result is that it means the summarization community has not yet found a reliable automatic metric that aligns with its research goal, to generate summaries with high-quality information. Then, we propose a simple and interpretable method of evaluating summaries which does directly measure information overlap and demonstrate how it can be used to gain insights into model behavior that could not be provided by other methods alone.