Adversarial Text Generation via Feature-Mover's Distance

TL;DR

This paper introduces a novel feature-mover's distance metric for GAN-based text generation, improving training stability and performance across various tasks by matching latent feature distributions of real and synthetic sentences.

Contribution

It proposes the feature-mover's distance, a new optimal transport-inspired metric, to enhance GAN training for text generation, addressing mode collapse and training brittleness.

Findings

Outperforms existing methods in unconditional text generation

Effective in style transfer from non-parallel text

Demonstrates success in unsupervised cipher cracking

Abstract

Generative adversarial networks (GANs) have achieved significant success in generating real-valued data. However, the discrete nature of text hinders the application of GAN to text-generation tasks. Instead of using the standard GAN objective, we propose to improve text-generation GAN via a novel approach inspired by optimal transport. Specifically, we consider matching the latent feature distributions of real and synthetic sentences using a novel metric, termed the feature-mover's distance (FMD). This formulation leads to a highly discriminative critic and easy-to-optimize objective, overcoming the mode-collapsing and brittle-training problems in existing methods. Extensive experiments are conducted on a variety of tasks to evaluate the proposed model empirically, including unconditional text generation, style transfer from non-parallel text, and unsupervised cipher cracking. The proposed model yields superior performance, demonstrating wide applicability and effectiveness.