Controllable Text Generation in the Instruction-Tuning Era

TL;DR

This paper evaluates controllable text generation methods in the context of instruction-tuned language models, revealing that prompt-based approaches often outperform traditional methods and introducing a new benchmark and dataset generation algorithm.

Contribution

It introduces ConGenBench, a comprehensive benchmark for controllable generation tasks, and proposes an algorithm for automatic constraint dataset creation using LLMs.

Findings

Prompt-based methods outperform traditional controllable methods on most tasks.

Prompt approaches match human performance on stylistic tasks.

The dataset generation algorithm expands research possibilities without pre-curated constraints.

Abstract

While most research on controllable text generation has focused on steering base Language Models, the emerging instruction-tuning and prompting paradigm offers an alternate approach to controllability. We compile and release ConGenBench, a testbed of 17 different controllable generation tasks, using a subset of it to benchmark the performance of 9 different baselines and methods on Instruction-tuned Language Models. To our surprise, we find that prompting-based approaches outperform controllable text generation methods on most datasets and tasks, highlighting a need for research on controllable text generation with Instruction-tuned Language Models in specific. Prompt-based approaches match human performance on most stylistic tasks while lagging on structural tasks, foregrounding a need to study more varied constraints and more challenging stylistic tasks. To facilitate such research, we provide an algorithm that uses only a task dataset and a Large Language Model with in-context capabilities to automatically generate a constraint dataset. This method eliminates the fields dependence on pre-curated constraint datasets, hence vastly expanding the range of constraints that can be studied in the future.