CASPR: Automated Evaluation Metric for Contrastive Summarization

TL;DR

CASPR is an automated, NLI-based metric designed to reliably evaluate the contrastiveness of summaries in contrastive summarization tasks, outperforming existing token-overlap and BERTScore baselines.

Contribution

This work introduces CASPR, a novel lightweight NLI-based evaluation metric that effectively measures contrastiveness in summaries without human input.

Findings

CASPR more reliably captures contrastiveness than baselines.

CASPR outperforms Distinctiveness Score and BERTScore on CoCoTRIP dataset.

The metric is simple, efficient, and sensitive to meaning-preserving lexical variations.

Abstract

Summarizing comparative opinions about entities (e.g., hotels, phones) from a set of source reviews, often referred to as contrastive summarization, can considerably aid users in decision making. However, reliably measuring the contrastiveness of the output summaries without relying on human evaluations remains an open problem. Prior work has proposed token-overlap based metrics, Distinctiveness Score, to measure contrast which does not take into account the sensitivity to meaning-preserving lexical variations. In this work, we propose an automated evaluation metric CASPR to better measure contrast between a pair of summaries. Our metric is based on a simple and light-weight method that leverages natural language inference (NLI) task to measure contrast by segmenting reviews into single-claim sentences and carefully aggregating NLI scores between them to come up with a summary-level score. We compare CASPR with Distinctiveness Score and a simple yet powerful baseline based on BERTScore. Our results on a prior dataset CoCoTRIP demonstrate that CASPR can more reliably capture the contrastiveness of the summary pairs compared to the baselines.