Deep State Space Models for Unconditional Word Generation

TL;DR

This paper introduces a non-autoregressive deep state space model for unconditional word generation that separates global and local uncertainties, achieving interpretability and competitive performance without autoregressive feedback.

Contribution

It presents a novel non-autoregressive model using flow-based variational inference, avoiding biases of autoregressive feedback and enhancing interpretability.

Findings

Achieves performance comparable to autoregressive models

Provides a highly interpretable generative framework

Eliminates need for annealing or auxiliary losses

Abstract

Autoregressive feedback is considered a necessity for successful unconditional text generation using stochastic sequence models. However, such feedback is known to introduce systematic biases into the training process and it obscures a principle of generation: committing to global information and forgetting local nuances. We show that a non-autoregressive deep state space model with a clear separation of global and local uncertainty can be built from only two ingredients: An independent noise source and a deterministic transition function. Recent advances on flow-based variational inference can be used to train an evidence lower-bound without resorting to annealing, auxiliary losses or similar measures. The result is a highly interpretable generative model on par with comparable auto-regressive models on the task of word generation.