On NMT Search Errors and Model Errors: Cat Got Your Tongue?

TL;DR

This paper investigates search and model errors in neural machine translation, revealing that beam search often fails to find optimal translations and that models tend to favor empty outputs due to length bias.

Contribution

It introduces an exact inference method combining beam and depth-first search to evaluate global model scores in NMT, exposing significant search and model errors.

Findings

Beam search fails to find global best scores in most cases.

Models often assign highest scores to empty translations.

Shorter translations are inherently favored due to length bias.

Abstract

We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.