Amortized Noisy Channel Neural Machine Translation

TL;DR

This paper explores creating an efficient amortized noisy channel neural machine translation model that achieves similar translation quality to traditional methods but with significantly faster inference, using three different training approaches.

Contribution

It introduces three novel training methods for amortized noisy channel NMT that improve inference speed while maintaining translation quality.

Findings

All approaches significantly speed up inference by 10-100 times.

Translation quality, measured by BLEU and BLEURT, remains comparable to BSR.

Reward metrics during decoding do not match BSR, indicating a trade-off.

Abstract

Noisy channel models have been especially effective in neural machine translation (NMT). However, recent approaches like "beam search and rerank" (BSR) incur significant computation overhead during inference, making real-world application infeasible. We aim to study if it is possible to build an amortized noisy channel NMT model such that when we do greedy decoding during inference, the translation accuracy matches that of BSR in terms of reward (based on the source-to-target log probability and the target-to-source log probability) and quality (based on BLEU and BLEURT). We attempt three approaches to train the new model: knowledge distillation, one-step-deviation imitation learning, and Q learning. The first approach obtains the noisy channel signal from a pseudo-corpus, and the latter two approaches aim to optimize toward a noisy-channel MT reward directly. For all three approaches, the generated translations fail to achieve rewards comparable to BSR, but the translation quality approximated by BLEU and BLEURT is similar to the quality of BSR-produced translations. Additionally, all three approaches speed up inference by 1-2 orders of magnitude.