Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

TL;DR

This paper introduces a unified framework for evaluating natural language generation by classifying tasks into compression, transduction, and creation, and developing interpretable, often reference-free metrics based on information alignment.

Contribution

It proposes a unifying perspective and a family of metrics for NLG evaluation based on information alignment, applicable across diverse tasks and properties.

Findings

Metrics show strong correlation with human judgment.

Applicable to summarization, style transfer, and dialog.

Operates often without gold reference data.

Abstract

Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective that facilitates the design of metrics for a wide range of language generation tasks and quality aspects. Based on the nature of information change from input to output, we classify NLG tasks into compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). The information alignment, or overlap, between input, context, and output text plays a common central role in characterizing the generation. Using the uniform concept of information alignment, we develop a family of interpretable metrics for various NLG tasks and aspects, often without need of gold reference data. To operationalize the metrics, we train self-supervised models to approximate information alignment as a prediction task. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog. With information alignment as the intermediate representation, we deliver a composable library for easy NLG evaluation and future metric design.