Evaluation of Text Generation: A Survey

TL;DR

This survey reviews recent evaluation methods for natural language generation, categorizing them into human-centric, automatic, and machine-learned metrics, highlighting progress, challenges, and future directions.

Contribution

It provides a comprehensive categorization and analysis of NLG evaluation methods, including case studies and future research proposals.

Findings

Progress in evaluation metrics for neural NLG models

Challenges in evaluating long and diverse text generation

Identification of promising future research directions

Abstract

The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two examples for task-specific NLG evaluations for automatic text summarization and long text generation, and conclude the paper by proposing future research directions.