Inducing Causal Structure for Abstractive Text Summarization

TL;DR

This paper introduces a causal modeling approach to abstractive text summarization, aiming to improve summary quality by focusing on underlying causal factors rather than spurious correlations.

Contribution

It proposes a Structural Causal Model and a causality-inspired Seq2Seq model to induce and utilize causal structures in summarization tasks.

Findings

The approach effectively identifies causal factors in summarization data.

Experimental results show improved performance on benchmark datasets.

Abstract

The mainstream of data-driven abstractive summarization models tends to explore the correlations rather than the causal relationships. Among such correlations, there can be spurious ones which suffer from the language prior learned from the training corpus and therefore undermine the overall effectiveness of the learned model. To tackle this issue, we introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We assume several latent causal factors and non-causal factors, representing the content and style of the document and summary. Theoretically, we prove that the latent factors in our SCM can be identified by fitting the observed training data under certain conditions. On the basis of this, we propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors, guiding us to pursue causal information for summary generation. The key idea is to reformulate the Variational Auto-encoder (VAE) to fit the joint distribution of the document and summary variables from the training corpus. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.