# A Test Suite and Manual Evaluation of Document-Level NMT at WMT19

## TL;DR

This paper introduces a test suite for evaluating document-level neural machine translation systems, focusing on discourse phenomena, and includes manual error analysis to identify relevant translation issues.

## Contribution

It provides a new test suite and manual evaluation methodology specifically designed for assessing discourse-related translation errors at the document level.

## Key findings

- Identified key translation errors affecting document-level NMT
- Provided a benchmark for future evaluation of discourse phenomena
- Manual analysis highlights common challenges in document translation

## Abstract

As the quality of machine translation rises and neural machine translation (NMT) is moving from sentence to document level translations, it is becoming increasingly difficult to evaluate the output of translation systems. We provide a test suite for WMT19 aimed at assessing discourse phenomena of MT systems participating in the News Translation Task. We have manually checked the outputs and identified types of translation errors that are relevant to document-level translation.