Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015

TL;DR

This paper demonstrates that neural reranking significantly enhances the subjective quality of machine translation outputs, especially in grammatical correctness, confirmed through both objective metrics and manual evaluation.

Contribution

It introduces a neural reranking component for syntax-based statistical machine translation and confirms its effectiveness in improving translation quality in both objective and manual assessments.

Findings

Neural reranking yields large BLEU score improvements.

Manual evaluations confirm quality gains from neural reranking.

Main improvements are in grammatical correctness, not lexical choice.

Abstract

This year, the Nara Institute of Science and Technology (NAIST)'s submission to the 2015 Workshop on Asian Translation was based on syntax-based statistical machine translation, with the addition of a reranking component using neural attentional machine translation models. Experiments re-confirmed results from previous work stating that neural MT reranking provides a large gain in objective evaluation measures such as BLEU, and also confirmed for the first time that these results also carry over to manual evaluation. We further perform a detailed analysis of reasons for this increase, finding that the main contributions of the neural models lie in improvement of the grammatical correctness of the output, as opposed to improvements in lexical choice of content words.