SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling

TL;DR

This paper introduces SueNes, a weakly supervised method for evaluating single-document summaries that does not rely on reference summaries and better captures linguistic quality, using corrupted references for training.

Contribution

The paper proposes a novel weakly supervised evaluation approach for summarization that leverages corrupted references, eliminating the need for reference summaries.

Findings

Outperforms baseline metrics in cross-domain tests

Better at assessing linguistic quality than existing metrics

Effective without reference summaries

Abstract

Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.