Parallel Refinements for Lexically Constrained Text Generation with BART

TL;DR

This paper introduces Constrained BART (CBART), a novel method for lexically constrained text generation that improves quality and diversity while reducing inference time by decomposing the task and enabling parallel token prediction.

Contribution

The paper proposes CBART, which transfers part of the generation process to the encoder and predicts tokens in parallel, enhancing efficiency and output quality in lexically constrained text generation.

Findings

CBART generates high-quality, diverse text with constraints.

CBART significantly reduces inference latency.

Experimental results outperform previous methods on benchmark datasets.

Abstract

Lexically constrained text generation aims to control the generated text by incorporating some pre-specified keywords into the output. Previous work injects lexical constraints into the output by controlling the decoding process or refining the candidate output iteratively, which tends to generate generic or ungrammatical sentences, and has high computational complexity. To address these challenges, we propose Constrained BART (CBART) for lexically constrained text generation. CBART leverages the pre-trained model BART and transfers part of the generation burden from the decoder to the encoder by decomposing this task into two sub-tasks, thereby improving the sentence quality. Concretely, we extend BART by adding a token-level classifier over the encoder, aiming at instructing the decoder where to replace and insert. Guided by the encoder, the decoder refines multiple tokens of the input in one step by inserting tokens before specific positions and re-predicting tokens with low confidence. To further reduce the inference latency, the decoder predicts all tokens in parallel. Experiment results on One-Billion-Word and Yelp show that CBART can generate plausible text with high quality and diversity while significantly accelerating inference.