Generating Sentences Using a Dynamic Canvas

TL;DR

This paper presents AUTR, a neural model that generates sentences through a dynamic attention mechanism and canvas memory, providing interpretability and competitive performance in sentence generation and reconstruction.

Contribution

Introduces AUTR, a novel recurrent neural network with dynamic attention and canvas memory for interpretable and efficient sentence generation.

Findings

AUTR learns meaningful latent sentence representations

Achieves competitive log-likelihood lower bounds

Effective at sentence generation, reconstruction, and missing word imputation

Abstract

We introduce the Attentive Unsupervised Text (W)riter (AUTR), which is a word level generative model for natural language. It uses a recurrent neural network with a dynamic attention and canvas memory mechanism to iteratively construct sentences. By viewing the state of the memory at intermediate stages and where the model is placing its attention, we gain insight into how it constructs sentences. We demonstrate that AUTR learns a meaningful latent representation for each sentence, and achieves competitive log-likelihood lower bounds whilst being computationally efficient. It is effective at generating and reconstructing sentences, as well as imputing missing words.