CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

TL;DR

This paper introduces a contrastive learning approach that improves the factual accuracy of abstractive summaries by training models to distinguish correct summaries from common error types, leading to more faithful outputs.

Contribution

The paper presents a novel contrastive learning framework utilizing reference and error-inducing summaries, with strategies tailored to common model errors, enhancing factuality in summarization.

Findings

Consistently produces more factual summaries across datasets.

Outperforms error correction and reranking methods.

Human evaluations confirm reduced errors in summaries.

Abstract

We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.