Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation

TL;DR

This study systematically compares human and automated evaluation methods for online review generation, revealing discrepancies and challenges in assessing the quality of neural language models.

Contribution

It provides a comprehensive large-scale analysis of evaluation metrics, highlighting the limitations of current automated methods and the complexity of human judgment in review generation.

Findings

Discriminative evaluators do not align well with human judgments.

Distinguishing machine-generated text remains difficult even for humans.

Lexical diversity correlates with evaluator assessments.

Abstract

We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machine-generated text can be distinguished from human-written text, as well as word overlap metrics that assess how similar the generated text compares to human-written references. We determine to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews. We find that human evaluators do not correlate well with discriminative evaluators, leaving a bigger question of whether adversarial accuracy is the correct objective for natural language generation. In general, distinguishing machine-generated text is challenging even for human evaluators, and human decisions correlate better with lexical overlaps. We find lexical diversity an intriguing metric that is indicative of the assessments of different evaluators. A post-experiment survey of participants provides insights into how to evaluate and improve the quality of natural language generation systems.