Text Infilling
Wanrong Zhu, Zhiting Hu, Eric Xing

TL;DR
This paper explores the general task of text infilling, filling multiple missing portions of text with variable lengths, and demonstrates that a self-attention model with segment-aware encoding significantly outperforms other methods.
Contribution
It introduces a comprehensive approach to text infilling with multiple missing segments and proposes a novel self-attention model that sets a new baseline.
Findings
Self-attention model outperforms other approaches
Created extensive supervised data for training and evaluation
Establishes a strong baseline for future research in text infilling
Abstract
Recent years have seen remarkable progress of text generation in different contexts, such as the most common setting of generating text from scratch, and the emerging paradigm of retrieval-and-rewriting. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life, yet is under-explored. Previous work has focused on restricted settings by either assuming single word per missing portion or limiting to a single missing portion to the end of the text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We study various approaches for the task, including a self-attention model with segment-aware position encoding and bidirectional context modeling. We create extensive supervised data by masking…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques
