Controllable Natural Language Generation with Contrastive Prefixes

TL;DR

This paper introduces a lightweight, controllable text generation framework using attribute-specific prefixes for GPT-2, enabling multi-aspect control with high linguistic quality through supervised and unsupervised training methods.

Contribution

It presents a novel prefix-based control method that considers relationships among prefixes and supports both single- and multi-aspect attribute control.

Findings

Effective control over generation attributes demonstrated

High linguistic quality maintained during controlled generation

Supports both supervised and unsupervised training approaches

Abstract

To guide the generation of large pretrained language models (LM), previous work has focused on directly fine-tuning the language model or utilizing an attribute discriminator. In this work, we propose a novel lightweight framework for controllable GPT2 generation, which utilizes a set of small attribute-specific vectors, called prefixes, to steer natural language generation. Different from prefix-tuning, where each prefix is trained independently, we take the relationship among prefixes into consideration and train multiple prefixes simultaneously. We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control while the combination of these two methods can achieve multi-aspect control. Experimental results on both single-aspect and multi-aspect control show that our methods can guide generation towards the desired attributes while keeping high linguistic quality.