Neural versus Phrase-Based Machine Translation Quality: a Case Study

TL;DR

This paper compares neural machine translation (NMT) and phrase-based machine translation (PBMT), analyzing their translation quality differences using professional post-edits, revealing strengths of NMT in modeling linguistic phenomena like verb reordering.

Contribution

It provides a detailed analysis of NMT versus PBMT outputs, highlighting linguistic aspects where NMT excels and identifying areas needing improvement.

Findings

NMT outperforms PBMT on English-German translation.

Neural models better handle verb reordering.

Some linguistic phenomena still challenge NMT.

Abstract

Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of-the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural versus phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models -- such as the reordering of verbs -- while pointing out other aspects that remain to be improved.