Does Summary Evaluation Survive Translation to Other Languages?

TL;DR

This paper investigates whether summary evaluation datasets created in one language can be reliably used in other languages through machine translation, finding limited transferability for most evaluation methods.

Contribution

The study evaluates the effectiveness of translating a summarization dataset into multiple languages and assesses the statistical equivalence of evaluation methods across translations.

Findings

Limited transferability of evaluation methods across languages

Some potential for dataset reuse in similar languages

Most evaluation methods are not statistically equivalent across translations

Abstract

The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.