# A Challenge Set Approach to Evaluating Machine Translation

## TL;DR

This paper introduces a challenge set method for evaluating machine translation systems by analyzing their ability to handle specific linguistic divergences, providing detailed insights into their strengths and remaining weaknesses.

## Contribution

It presents a novel challenge set approach for detailed error analysis in machine translation, focusing on structural divergences between languages.

## Key findings

- Neural machine translation outperforms phrase-based systems on many linguistic phenomena.
- Certain complex linguistic divergences remain challenging for neural systems.
- The approach offers a more nuanced evaluation of translation quality.

## Abstract

Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to translation evaluation and error analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system's capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English-French challenge set, and use it to analyze phrase-based and neural systems. The resulting analysis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phenomena remain out of reach.