Control Prefixes for Parameter-Efficient Text Generation

TL;DR

This paper introduces Control Prefixes, a dynamic prompt tuning method that incorporates attribute-level information into language models, enabling more controlled and efficient text generation with state-of-the-art results.

Contribution

It proposes a novel dynamic prompt tuning approach that integrates attribute-level representations, outperforming full fine-tuning on multiple natural language generation datasets.

Findings

Control Prefixes outperform full fine-tuning methods.

Achieves state-of-the-art results on WebNLG and other datasets.

Enables conditional, controlled text generation with fewer parameters.

Abstract

Prefix-tuning is a powerful lightweight technique for adapting a large pre-trained language model to a downstream application. However, it uses the same dataset-level tuned prompt for all examples in the dataset. We extend this idea and propose a dynamic method, Control Prefixes, which allows for the inclusion of conditional input-dependent information, combining the benefits of prompt tuning and controlled generation. The method incorporates attribute-level learnable representations into different layers of a pre-trained transformer, allowing for the generated text to be guided in a particular direction. We provide a systematic evaluation of the technique and apply it to five datasets from the GEM benchmark for natural language generation (NLG). Although the aim is to develop a parameter-efficient model, we show Control Prefixes can even outperform full fine-tuning methods. We present state-of-the-art results on several data-to-text datasets, including WebNLG.