Autoregressive Text Generation Beyond Feedback Loops

TL;DR

This paper introduces a hybrid model combining latent state space and CRF observation models to improve autoregressive text generation, reducing biases and enhancing performance over traditional RNN and GAN methods.

Contribution

It proposes a novel autoregressive observation model that balances local word correlations with non-autoregressive state evolution, addressing train-test bias issues.

Findings

Outperforms RNN and GAN baselines in sentence generation

Reduces biases from train-test discrepancies

Avoids common failure modes of autoregressive models

Abstract

Autoregressive state transitions, where predictions are conditioned on past predictions, are the predominant choice for both deterministic and stochastic sequential models. However, autoregressive feedback exposes the evolution of the hidden state trajectory to potential biases from well-known train-test discrepancies. In this paper, we combine a latent state space model with a CRF observation model. We argue that such autoregressive observation models form an interesting middle ground that expresses local correlations on the word level but keeps the state evolution non-autoregressive. On unconditional sentence generation we show performance improvements compared to RNN and GAN baselines while avoiding some prototypical failure modes of autoregressive models.