Distributional Discrepancy: A Metric for Unconditional Text Generation

TL;DR

This paper introduces a new distributional discrepancy metric for evaluating unconditional text generation, addressing the challenge of balancing quality and diversity in model assessment.

Contribution

It proposes a neural-network-based method to estimate the distributional discrepancy, improving model ranking over existing metrics.

Findings

DD outperforms existing metrics in ranking generative models

The method effectively evaluates both syntactic and real data

DD provides a more consistent assessment of quality and diversity

Abstract

The purpose of unconditional text generation is to train a model with real sentences, then generate novel sentences of the same quality and diversity as the training data. However, when different metrics are used for comparing the methods of unconditional text generation, contradictory conclusions are drawn. The difficulty is that both the diversity and quality of the sample should be considered simultaneously when the models are evaluated. To solve this problem, a novel metric of distributional discrepancy (DD) is designed to evaluate generators based on the discrepancy between the generated and real training sentences. However, it cannot compute the DD directly because the distribution of real sentences is unavailable. Thus, we propose a method for estimating the DD by training a neural-network-based text classifier. For comparison, three existing metrics, bi-lingual evaluation understudy (BLEU) versus self-BLEU, language model score versus reverse language model score, and Fr\'{e}chet embedding distance, along with the proposed DD, are used to evaluate two popular generative models of long short-term memory and generative pretrained transformer 2 on both syntactic and real data. Experimental results show that DD is significantly better than the three existing metrics for ranking these generative models.