Analyzing Uncertainty in Neural Machine Translation

TL;DR

This paper investigates the inherent and extrinsic uncertainties in neural machine translation, proposing new tools and metrics to evaluate and improve model calibration and translation diversity.

Contribution

It introduces novel methods for assessing uncertainty in NMT models and provides practical tools to enhance translation quality and diversity.

Findings

Search strategies are effective despite models spreading probability mass too broadly.

Models tend to underestimate rare words and lack diversity in translations.

Tools for assessing and fixing model calibration are proposed.

Abstract

Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. As part of this study, we release multiple human reference translations for two popular benchmarks.