A Distributional Approach to Controlled Text Generation

TL;DR

This paper introduces a novel distributional framework for controlled text generation using pre-trained language models, enabling flexible pointwise and distributional constraints while minimizing divergence from the original model.

Contribution

It presents the first model capable of handling both pointwise and distributional constraints within a unified formal framework, utilizing an explicit energy-based model and adaptive policy gradient training.

Findings

Outperforms baselines in balancing constraint satisfaction and divergence

Demonstrates potential to mitigate bias in language models

Shows faster convergence with adaptive techniques

Abstract

We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both "pointwise" and "distributional" constraints over the target LM -- to our knowledge, the first model with such generality -- while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence. (Code available at https://github.com/naver/gdc)