The Detection of Distributional Discrepancy for Text Generation

TL;DR

This paper introduces two new metrics to measure how much the distribution of generated text differs from real text, revealing that current language GANs fail to reduce this discrepancy despite multiple training rounds.

Contribution

The paper proposes two theoretical metrics for accurately measuring distributional discrepancy in text generation and evaluates their effectiveness in assessing language GANs.

Findings

The discrepancy between real and generated text remains large after training.

Existing language GANs do not effectively minimize distributional differences.

Discrepancy increases with more adversarial training rounds.

Abstract

The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.