MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

TL;DR

MoverScore is a new evaluation metric for text generation that uses contextualized embeddings and Earth Mover Distance to better align with human judgment across various tasks.

Contribution

The paper introduces MoverScore, a novel metric combining contextualized embeddings with Earth Mover Distance, showing improved correlation with human assessments.

Findings

Metrics with contextualized embeddings outperform traditional ones.

MoverScore generalizes well across multiple text generation tasks.

The metric is accessible via a web service for practical use.

Abstract

A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.