Evaluating Text GANs as Language Models

TL;DR

This paper introduces a method to evaluate text-generating GANs using traditional language model metrics, revealing they currently underperform compared to state-of-the-art language models, and aims to improve understanding and development of GAN-based text generation.

Contribution

It proposes an approximation method to evaluate GANs with LM metrics, enabling better comparison and understanding of GANs versus traditional language models.

Findings

GANs perform substantially worse than state-of-the-art LMs

The evaluation method facilitates understanding of GANs' potential

The approach promotes progress in GAN-based text generation

Abstract

Generative Adversarial Networks (GANs) are a promising approach for text generation that, unlike traditional language models (LM), does not suffer from the problem of ``exposure bias''. However, A major hurdle for understanding the potential of GANs for text generation is the lack of a clear evaluation metric. In this work, we propose to approximate the distribution of text generated by a GAN, which permits evaluating them with traditional probability-based LM metrics. We apply our approximation procedure on several GAN-based models and show that they currently perform substantially worse than state-of-the-art LMs. Our evaluation procedure promotes better understanding of the relation between GANs and LMs, and can accelerate progress in GAN-based text generation.