Correcting Length Bias in Neural Machine Translation

TL;DR

This paper investigates length bias issues in neural machine translation, demonstrating that correcting brevity problems can improve beam search performance, and proposes a simple, effective method for tuning translation length.

Contribution

It introduces a straightforward approach to correct length bias in NMT, linking it to beam search problems and providing a practical tuning method.

Findings

Correcting brevity bias improves beam search results.

A simple per-word reward effectively addresses length issues.

Perceptron-based tuning is quick and effective.

Abstract

We study two problems in neural machine translation (NMT). First, in beam search, whereas a wider beam should in principle help translation, it often hurts NMT. Second, NMT has a tendency to produce translations that are too short. Here, we argue that these problems are closely related and both rooted in label bias. We show that correcting the brevity problem almost eliminates the beam problem; we compare some commonly-used methods for doing this, finding that a simple per-word reward works well; and we introduce a simple and quick way to tune this reward using the perceptron algorithm.