Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation

TL;DR

This paper introduces Constrained Abstractive Summarization (CAS), a method that uses token constraints during decoding to improve factual accuracy and lexical overlap in summaries, validated through experiments on benchmark datasets.

Contribution

The paper proposes a novel constrained decoding framework for abstractive summarization that enhances factual consistency and allows for human-guided interactive summarization.

Findings

CAS improves ROUGE scores and factual consistency.

Automatic constraints from keyphrases boost summary quality.

Manual constraints significantly enhance ROUGE-2 scores.

Abstract

Despite significant progress, state-of-the-art abstractive summarization methods are still prone to hallucinate content inconsistent with the source document. In this paper, we propose Constrained Abstractive Summarization (CAS), a general setup that preserves the factual consistency of abstractive summarization by specifying tokens as constraints that must be present in the summary. We adopt lexically constrained decoding, a technique generally applicable to autoregressive generative models, to fulfill CAS and conduct experiments in two scenarios: (1) automatic summarization without human involvement, where keyphrases are extracted from the source document and used as constraints; (2) human-guided interactive summarization, where human feedback in the form of manual constraints are used to guide summary generation. Automatic and human evaluations on two benchmark datasets demonstrate that CAS improves both lexical overlap (ROUGE) and factual consistency of abstractive summarization. In particular, we observe up to 13.8 ROUGE-2 gains when only one manual constraint is used in interactive summarization.