Language Modeling with Generative Adversarial Networks

TL;DR

This paper explores the challenges of applying GANs to language generation, compares different training approaches, and finds that weaker regularization improves WGAN training stability.

Contribution

It provides a comparative analysis of training methods for GANs in language modeling and demonstrates improved convergence with weaker Lipschitz regularization.

Findings

WGANs with weaker regularization converge better.

Pre-training with maximum-likelihood affects GAN training.

Comparison of training approaches for language GANs.

Abstract

Generative Adversarial Networks (GANs) have been promising in the field of image generation, however, they have been hard to train for language generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them which causes high levels of instability in training GANs. Consequently, past work has resorted to pre-training with maximum-likelihood or training GANs without pre-training with a WGAN objective with a gradient penalty. In this study, we present a comparison of those approaches. Furthermore, we present the results of some experiments that indicate better training and convergence of Wasserstein GANs (WGANs) when a weaker regularization term is enforcing the Lipschitz constraint.