Communication-based Evaluation for Natural Language Generation

TL;DR

This paper proposes a communication-based evaluation method for natural language generation that assesses how effectively an NLG system conveys information, using the Rational Speech Acts model, and demonstrates better alignment with quality categories than traditional n-gram overlap metrics.

Contribution

It introduces a novel communication-focused evaluation framework for NLG based on pragmatic language use, improving alignment with perceived quality over existing metrics.

Findings

Communication-based evaluation correlates better with quality categories.

Method outperforms n-gram overlap measures in conveying information.

Demonstrated on a color reference dataset.

Abstract

Natural language generation (NLG) systems are commonly evaluated using n-gram overlap measures (e.g. BLEU, ROUGE). These measures do not directly capture semantics or speaker intentions, and so they often turn out to be misaligned with our true goals for NLG. In this work, we argue instead for communication-based evaluations: assuming the purpose of an NLG system is to convey information to a reader/listener, we can directly evaluate its effectiveness at this task using the Rational Speech Acts model of pragmatic language use. We illustrate with a color reference dataset that contains descriptions in pre-defined quality categories, showing that our method better aligns with these quality categories than do any of the prominent n-gram overlap methods.