Analyzing Neural MT Search and Model Performance

TL;DR

This paper analyzes whether current search algorithms and model complexities are sufficient in neural machine translation, finding that existing search methods are adequate and small n-best lists contain high-quality translations.

Contribution

It separates search and modeling effects in NMT, demonstrating that current search algorithms are sufficient and small n-best lists can yield high-quality translations.

Findings

Better translations are already in the search space of less performant systems.

Current search algorithms are sufficient for NMT.

Small n-best lists of 50 hypotheses contain notably better translations.

Abstract

In this paper, we offer an in-depth analysis about the modeling and search performance. We address the question if a more complex search algorithm is necessary. Furthermore, we investigate the question if more complex models which might only be applicable during rescoring are promising. By separating the search space and the modeling using $n$-best list reranking, we analyze the influence of both parts of an NMT system independently. By comparing differently performing NMT systems, we show that the better translation is already in the search space of the translation systems with less performance. This results indicate that the current search algorithms are sufficient for the NMT systems. Furthermore, we could show that even a relatively small $n$-best list of $50$ hypotheses already contain notably better translations.