HLT-MT: High-resource Language-specific Training for Multilingual Neural Machine Translation

TL;DR

This paper introduces HLT-MT, a two-stage training approach with language-specific modules to improve multilingual neural machine translation, especially reducing negative interference among languages, leading to better performance on benchmarks.

Contribution

The paper proposes a novel high-resource language-specific training method with a two-stage process and language-specific modules to enhance multilingual translation quality.

Findings

Outperforms strong baselines on WMT-10 and OPUS-100 benchmarks.

Effectively mitigates negative interference in multilingual training.

Improves translation quality for high-resource and low-resource languages.

Abstract

Multilingual neural machine translation (MNMT) trained in multiple language pairs has attracted considerable attention due to fewer model parameters and lower training costs by sharing knowledge among multiple languages. Nonetheless, multilingual training is plagued by language interference degeneration in shared parameters because of the negative interference among different translation directions, especially on high-resource languages. In this paper, we propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference, which adopts the two-stage training with the language-specific selection mechanism. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder to enhance the translation quality of high-resource directions. Next, the model is further trained on all available corpora to transfer knowledge from high-resource languages (HRLs) to low-resource languages (LRLs). Experimental results show that HLT-MT outperforms various strong baselines on WMT-10 and OPUS-100 benchmarks. Furthermore, the analytic experiments validate the effectiveness of our method in mitigating the negative interference in multilingual training.