Text Infilling

TL;DR

This paper explores the general task of text infilling, filling multiple missing portions of text with variable lengths, and demonstrates that a self-attention model with segment-aware encoding significantly outperforms other methods.

Contribution

It introduces a comprehensive approach to text infilling with multiple missing segments and proposes a novel self-attention model that sets a new baseline.

Findings

Self-attention model outperforms other approaches

Created extensive supervised data for training and evaluation

Establishes a strong baseline for future research in text infilling

Abstract

Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking out text with varying strategies. Experiments show the self-attention model greatly outperforms others, creating a strong baseline for future research.