Causal Intervention for Abstractive Related Work Generation

TL;DR

This paper introduces a causal intervention module for abstractive related work generation, enhancing the coherence and quality of generated texts by modeling and intervening in causal relations within the generation process.

Contribution

It proposes a novel CaM module that captures causal relations using do-calculus and integrates it with Transformers for improved related work generation.

Findings

Causal intervention improves generation coherence.

CaM outperforms baseline models on real-world datasets.

Enhanced causal modeling leads to higher quality related work texts.

Abstract

Abstractive related work generation has attracted increasing attention in generating coherent related work that better helps readers grasp the background in the current research. However, most existing abstractive models ignore the inherent causality of related work generation, leading to low quality of generated related work and spurious correlations that affect the models' generalizability. In this study, we argue that causal intervention can address these limitations and improve the quality and coherence of the generated related works. To this end, we propose a novel Causal Intervention Module for Related Work Generation (CaM) to effectively capture causalities in the generation process and improve the quality and coherence of the generated related works. Specifically, we first model the relations among sentence order, document relation, and transitional content in related work generation using a causal graph. Then, to implement the causal intervention and mitigate the negative impact of spurious correlations, we use do-calculus to derive ordinary conditional probabilities and identify causal effects through CaM. Finally, we subtly fuse CaM with Transformer to obtain an end-to-end generation model. Extensive experiments on two real-world datasets show that causal interventions in CaM can effectively promote the model to learn causal relations and produce related work of higher quality and coherence.