Language Generation with Recurrent Generative Adversarial Networks without Pre-training

TL;DR

This paper demonstrates that recurrent neural networks can be trained from scratch with GANs for language generation using curriculum learning, significantly improving sequence quality without pre-training.

Contribution

It introduces a curriculum learning approach enabling training of RNN-based GANs for text generation from scratch, bypassing the need for pre-training.

Findings

Recurrent GANs trained with curriculum learning produce higher quality text sequences.

The approach outperforms convolutional baselines in sequence quality.

Training from scratch is feasible for language GANs with the proposed method.

Abstract

Generative Adversarial Networks (GANs) have shown great promise recently in image generation. Training GANs for language generation has proven to be more difficult, because of the non-differentiable nature of generating text with recurrent neural networks. Consequently, past work has either resorted to pre-training with maximum-likelihood or used convolutional networks for generation. In this work, we show that recurrent neural networks can be trained to generate text with GANs from scratch using curriculum learning, by slowly teaching the model to generate sequences of increasing and variable length. We empirically show that our approach vastly improves the quality of generated sequences compared to a convolutional baseline.