A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy

TL;DR

This paper introduces a novel, training-free, reference-free summarization evaluation metric that combines centrality-weighted relevance and self-referenced redundancy scores, outperforming existing methods.

Contribution

It proposes a new evaluation metric that does not require training or references, using centrality-weighted relevance and self-redundancy to assess summaries.

Findings

Significantly outperforms existing evaluation methods.

Effective on both multi-document and single-document summarization.

Utilizes a novel combination of relevance and redundancy scores.

Abstract

In recent years, reference-based and supervised summarization evaluation metrics have been widely explored. However, collecting human-annotated references and ratings are costly and time-consuming. To avoid these limitations, we propose a training-free and reference-free summarization evaluation metric. Our metric consists of a centrality-weighted relevance score and a self-referenced redundancy score. The relevance score is computed between the pseudo reference built from the source document and the given summary, where the pseudo reference content is weighted by the sentence centrality to provide importance guidance. Besides an $F_1$-based relevance score, we also design an $F_\beta$-based variant that pays more attention to the recall score. As for the redundancy score of the summary, we compute a self-masked similarity score with the summary itself to evaluate the redundant information in the summary. Finally, we combine the relevance and redundancy scores to produce the final evaluation score of the given summary. Extensive experiments show that our methods can significantly outperform existing methods on both multi-document and single-document summarization evaluation.