Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation

TL;DR

This paper introduces an automatic method to extract challenge sets focusing on long-distance dependencies in neural machine translation, providing a scalable way to evaluate model performance on complex syntactic phenomena.

Contribution

It proposes a novel automatic approach for creating large challenge sets that target long-distance dependencies in translation, complementing existing evaluation metrics.

Findings

Transformer MT models are less biased towards monotonic reordering.

Long-distance dependencies remain challenging for current models.

The extracted challenge sets enable reliable automatic evaluation.

Abstract

We show that the state of the art Transformer Machine Translation (MT) model is not biased towards monotonic reordering (unlike previous recurrent neural network models), but that nevertheless, long-distance dependencies remain a challenge for the model. Since most dependencies are short-distance, common evaluation metrics will be little influenced by how well systems perform on them. We, therefore, propose an automatic approach for extracting challenge sets replete with long-distance dependencies and argue that evaluation using this methodology provides a complementary perspective on system performance. To support our claim, we compile challenge sets for English-German and German-English, which are much larger than any previously released challenge set for MT. The extracted sets are large enough to allow reliable automatic evaluation, which makes the proposed approach a scalable and practical solution for evaluating MT performance on the long-tail of syntactic phenomena.