The Case for a Single Model that can Both Generate Continuations and Fill in the Blank

TL;DR

This paper demonstrates that a single pre-trained language model can effectively perform both fill-in-the-blank and continuation tasks, offering versatile text generation capabilities with controllable output features.

Contribution

The study shows that models trained with a fill-in-the-blank objective can handle both tasks, unlike continuation-only models, and can be fine-tuned for detailed control over generated text.

Findings

FitB-trained models perform well on both tasks

Continuation-trained models do not handle FitB effectively

FitB models can be fine-tuned for control over length and word choice

Abstract

The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform _both_ FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.