What Level of Quality can Neural Machine Translation Attain on Literary Text?

TL;DR

This study evaluates neural machine translation's ability to translate literary texts, specifically novels, demonstrating significant improvements over traditional methods and showing that NMT can produce translations comparable to professional human work.

Contribution

First large-scale comparison of NMT and PBSMT on literary texts, showing NMT's superior performance and potential for high-quality literary translation.

Findings

NMT outperforms PBSMT with an 11% relative BLEU score improvement.

Human evaluation shows NMT translations are often perceived as equivalent to professional translations.

Training on large literary datasets enables NMT to better handle complex literary language.

Abstract

Given the rise of a new approach to MT, Neural MT (NMT), and its promising performance on different text types, we assess the translation quality it can attain on what is perceived to be the greatest challenge for MT: literary text. Specifically, we target novels, arguably the most popular type of literary text. We build a literary-adapted NMT system for the English-to-Catalan translation direction and evaluate it against a system pertaining to the previous dominant paradigm in MT: statistical phrase-based MT (PBSMT). To this end, for the first time we train MT systems, both NMT and PBSMT, on large amounts of literary text (over 100 million words) and evaluate them on a set of twelve widely known novels spanning from the the 1920s to the present day. According to the BLEU automatic evaluation metric, NMT is significantly better than PBSMT (p < 0.01) on all the novels considered. Overall, NMT results in a 11% relative improvement (3 points absolute) over PBSMT. A complementary human evaluation on three of the books shows that between 17% and 34% of the translations, depending on the book, produced by NMT (versus 8% and 20% with PBSMT) are perceived by native speakers of the target language to be of equivalent quality to translations produced by a professional human translator.