Generating Sentences by Editing Prototypes

TL;DR

This paper introduces a novel sentence generation model that creates sentences by editing a prototype, leading to better language modeling perplexity and more interpretable semantics compared to traditional methods.

Contribution

The paper presents a prototype-then-edit generative model that improves language modeling and provides interpretable latent edit vectors for semantic understanding.

Findings

Improved perplexity in language modeling

Higher quality sentence generation according to human evaluation

Latent edit vectors capture interpretable semantics

Abstract

We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perplexity on language modeling and generates higher quality outputs according to human evaluation. Furthermore, the model gives rise to a latent edit vector that captures interpretable semantics such as sentence similarity and sentence-level analogies.