SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder

TL;DR

SummScore is a new comprehensive evaluation metric for summarization quality that leverages CrossEncoder to better align with human judgments, capturing subtle semantic differences and evaluating multiple quality aspects.

Contribution

The paper introduces SummScore, a novel evaluation metric using CrossEncoder with four submodels, improving correlation with human scores and addressing diversity issues in summary evaluation.

Findings

SummScore outperforms existing metrics in correlation with human scores.

It effectively measures coherence, consistency, fluency, and relevance.

SummScore provides detailed evaluation results for 16 summarization models.

Abstract

Text summarization models are often trained to produce summaries that meet human quality requirements. However, the existing evaluation metrics for summary text are only rough proxies for summary quality, suffering from low correlation with human scoring and inhibition of summary diversity. To solve these problems, we propose SummScore, a comprehensive metric for summary quality evaluation based on CrossEncoder. Firstly, by adopting the original-summary measurement mode and comparing the semantics of the original text, SummScore gets rid of the inhibition of summary diversity. With the help of the text-matching pre-training Cross-Encoder, SummScore can effectively capture the subtle differences between the semantics of summaries. Secondly, to improve the comprehensiveness and interpretability, SummScore consists of four fine-grained submodels, which measure Coherence, Consistency, Fluency, and Relevance separately. We use semi-supervised multi-rounds of training to improve the performance of our model on extremely limited annotated data. Extensive experiments show that SummScore significantly outperforms existing evaluation metrics in the above four dimensions in correlation with human scoring. We also provide the quality evaluation results of SummScore on 16 mainstream summarization models for later research.