A Causal Lens for Controllable Text Generation

TL;DR

This paper introduces a causal framework for controllable text generation that unifies attribute-conditional generation and text attribute transfer, effectively reducing bias and enhancing control accuracy.

Contribution

It formulates controllable text generation within a causal inference framework, addressing biases and unifying two key tasks under a single principled approach.

Findings

Causal approach outperforms previous models in control accuracy.

Significant reduction in generation bias achieved.

Effective even with limited confounding factor data.

Abstract

Controllable text generation concerns two fundamental tasks of wide applications, namely generating text of given attributes (i.e., attribute-conditional generation), and minimally editing existing text to possess desired attributes (i.e., text attribute transfer). Extensive prior work has largely studied the two problems separately, and developed different conditional models which, however, are prone to producing biased text (e.g., various gender stereotypes). This paper proposes to formulate controllable text generation from a principled causal perspective which models the two tasks with a unified framework. A direct advantage of the causal formulation is the use of rich causality tools to mitigate generation biases and improve control. We treat the two tasks as interventional and counterfactual causal inference based on a structural causal model, respectively. We then apply the framework to the challenging practical setting where confounding factors (that induce spurious correlations) are observable only on a small fraction of data. Experiments show significant superiority of the causal approach over previous conditional models for improved control accuracy and reduced bias.