Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Level

TL;DR

This paper investigates the effectiveness of automatic evaluation metrics for longer paragraph translations in machine translation, proposing a new dataset creation method and benchmarking metrics at the paragraph level.

Contribution

It introduces a method for creating paragraph-level evaluation datasets from sentence-level data and benchmarks existing metrics, revealing that sentence-level metrics perform well at paragraph evaluation.

Findings

Sentence-level metrics are as effective as paragraph-specific metrics for evaluating paragraphs.

A new dataset creation method for paragraph-level evaluation is proposed.

Limitations in datasets may affect the assessment of paragraph translation phenomena.

Abstract

As research on machine translation moves to translating text beyond the sentence level, it remains unclear how effective automatic evaluation metrics are at scoring longer translations. In this work, we first propose a method for creating paragraph-level data for training and meta-evaluating metrics from existing sentence-level data. Then, we use these new datasets to benchmark existing sentence-level metrics as well as train learned metrics at the paragraph level. Interestingly, our experimental results demonstrate that using sentence-level metrics to score entire paragraphs is equally as effective as using a metric designed to work at the paragraph level. We speculate this result can be attributed to properties of the task of reference-based evaluation as well as limitations of our datasets with respect to capturing all types of phenomena that occur in paragraph-level translations.