Discourse Cohesion Evaluation for Document-Level Neural Machine Translation

TL;DR

This paper introduces DCoEM, a new evaluation method for document-level neural machine translation that assesses discourse cohesion across four manners, addressing the limitations of traditional sentence-level metrics like BLEU.

Contribution

The paper presents a novel discourse cohesion evaluation method and a comprehensive test suite to better measure document-level translation quality.

Findings

DCoEM effectively evaluates discourse cohesion in document translations.

The test suite covers four cohesive manners: reference, conjunction, substitution, and lexical cohesion.

Results show DCoEM's practicality and importance in assessing document-level NMT performance.

Abstract

It is well known that translations generated by an excellent document-level neural machine translation (NMT) model are consistent and coherent. However, existing sentence-level evaluation metrics like BLEU can hardly reflect the model's performance at the document level. To tackle this issue, we propose a Discourse Cohesion Evaluation Method (DCoEM) in this paper and contribute a new test suite that considers four cohesive manners (reference, conjunction, substitution, and lexical cohesion) to measure the cohesiveness of document translations. The evaluation results on recent document-level NMT systems show that our method is practical and essential in estimating translations at the document level.