Continual Knowledge Distillation for Neural Machine Translation

TL;DR

This paper introduces a continual knowledge distillation method that leverages existing translation models to enhance a target model, demonstrating significant improvements in Chinese-English and German-English translation tasks.

Contribution

The paper proposes a novel continual knowledge distillation approach that sequentially transfers knowledge from multiple trained models to improve neural machine translation.

Findings

Significant improvements over strong baselines

Robust to malicious models

Effective in both homogeneous and heterogeneous settings

Abstract

While many parallel corpora are not publicly accessible for data copyright, data privacy and competitive differentiation reasons, trained translation models are increasingly available on open platforms. In this work, we propose a method called continual knowledge distillation to take advantage of existing translation models to improve one model of interest. The basic idea is to sequentially transfer knowledge from each trained model to the distilled model. Extensive experiments on Chinese-English and German-English datasets show that our method achieves significant and consistent improvements over strong baselines under both homogeneous and heterogeneous trained model settings and is robust to malicious models.