Unifying Human and Statistical Evaluation for Natural Language Generation

TL;DR

This paper introduces HUSE, a unified evaluation framework combining human and statistical methods to assess both quality and diversity in natural language generation, revealing limitations of existing metrics.

Contribution

The paper proposes a novel evaluation metric, HUSE, that unifies human and statistical assessments to better measure quality and diversity in NLG systems.

Findings

HUSE detects diversity issues overlooked by human evaluation.

Improving quality via annealing can reduce diversity as measured by HUSE.

HUSE effectively identifies models that plagiarize from training data.

Abstract

How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is human- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity.