UM4: Unified Multilingual Multiple Teacher-Student Model for Zero-Resource Neural Machine Translation

TL;DR

UM4 introduces a unified teacher-student framework for zero-resource neural machine translation, effectively leveraging multiple teachers and monolingual data to improve translation quality across 72 language directions.

Contribution

The paper proposes a novel unified multi-teacher approach that combines source, target, and pivot teachers to enhance zero-resource NMT performance.

Findings

Outperforms previous methods on WMT benchmark

Effective utilization of monolingual data via pivot-teacher

Significant improvements across 72 translation directions

Abstract

Most translation tasks among languages belong to the zero-resource translation problem where parallel corpora are unavailable. Multilingual neural machine translation (MNMT) enables one-pass translation using shared semantic space for all languages compared to the two-pass pivot translation but often underperforms the pivot-based method. In this paper, we propose a novel method, named as Unified Multilingual Multiple teacher-student Model for NMT (UM4). Our method unifies source-teacher, target-teacher, and pivot-teacher models to guide the student model for the zero-resource translation. The source teacher and target teacher force the student to learn the direct source to target translation by the distilled knowledge on both source and target sides. The monolingual corpus is further leveraged by the pivot-teacher model to enhance the student model. Experimental results demonstrate that our model of 72 directions significantly outperforms previous methods on the WMT benchmark.