Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generation

TL;DR

Tailor introduces a prompt-based method for attribute-controlled text generation that efficiently combines pre-trained attribute prompts without retraining, enhancing multi-attribute generation while maintaining fluency.

Contribution

The paper proposes a novel prompt-based framework with multi-attribute prompt masking and a trainable prompt connector, enabling effective multi-attribute text generation without additional training.

Findings

Strong performance on 11 attribute-specific tasks

Effective multi-attribute generation with minimal training parameters

Maintains fluency and attribute control in generated text

Abstract

Attribute-based Controlled Text Generation (CTG) refers to generating sentences that satisfy desirable attributes (e.g., emotions and topics). Existing works often utilize fine-tuning or resort to extra attribute classifiers, yet suffer from storage and inference time increases. To address these concerns, we explore attribute-based CTG in a prompt-based manner. In short, the proposed Tailor represents each attribute as a pre-trained continuous vector (i.e., single-attribute prompt) and guides the generation of a fixed PLM switch to a pre-specified attribute. We experimentally find that these prompts can be simply concatenated as a whole to multi-attribute CTG without any re-training, yet raises problems of fluency decrease and position sensitivity. To this end, Tailor provides a multi-attribute prompt mask and a re-indexing position-ids sequence to bridge the gap between the training (one prompt for each task) and testing stage (concatenating more than one prompt). To further enhance such single-attribute prompt combinations, Tailor also introduces a trainable prompt connector, which can be concatenated with any two single-attribute prompts to multi-attribute text generation. Experiments on 11 attribute-specific generation tasks demonstrate strong performances of Tailor on both single-attribute and multi-attribute CTG, with 0.08\% training parameters of a GPT-2.