One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation

TL;DR

This paper introduces DDRS, a method for non-autoregressive translation that uses multiple references and reference selection to better address the multi-modality problem, leading to improved translation quality.

Contribution

It proposes a novel diverse distillation approach with reference selection, generating multiple high-quality references and selecting the best match during training to enhance NAT performance.

Findings

Achieves 29.82 BLEU on WMT14 En-De with one decoding pass

Outperforms previous NAT models by over 1 BLEU

Demonstrates effectiveness of multiple references and reference selection

Abstract

Non-autoregressive neural machine translation (NAT) suffers from the multi-modality problem: the source sentence may have multiple correct translations, but the loss function is calculated only according to the reference sentence. Sequence-level knowledge distillation makes the target more deterministic by replacing the target with the output from an autoregressive model. However, the multi-modality problem in the distilled dataset is still nonnegligible. Furthermore, learning from a specific teacher limits the upper bound of the model capability, restricting the potential of NAT models. In this paper, we argue that one reference is not enough and propose diverse distillation with reference selection (DDRS) for NAT. Specifically, we first propose a method called SeedDiv for diverse machine translation, which enables us to generate a dataset containing multiple high-quality reference translations for each source sentence. During the training, we compare the NAT output with all references and select the one that best fits the NAT output to train the model. Experiments on widely-used machine translation benchmarks demonstrate the effectiveness of DDRS, which achieves 29.82 BLEU with only one decoding pass on WMT14 En-De, improving the state-of-the-art performance for NAT by over 1 BLEU. Source code: https://github.com/ictnlp/DDRS-NAT