Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models

TL;DR

This paper introduces a domain adaptation method for pre-trained multilingual NMT models that enhances performance on specialized data while preserving generic domain quality across multiple language pairs.

Contribution

It proposes a novel fine-tuning procedure combining embeddings freezing and adversarial loss for effective domain adaptation in mNMT models.

Findings

+10 BLEU on specialized data

Minimal BLEU loss on generic data (-0.01 to -0.5)

Improves industry applicability of mNMT models

Abstract

Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to find in all language pairs. Therefore, fine-tuning a mNMT model on a specialized domain is hard. In this context, we decided to focus on a new task: Domain Adaptation of a pre-trained mNMT model on a single pair of language while trying to maintain model quality on generic domain data for all language pairs. The risk of loss on generic domain and on other pairs is high. This task is key for mNMT model adoption in the industry and is at the border of many others. We propose a fine-tuning procedure for the generic mNMT that combines embeddings freezing and adversarial loss. Our experiments demonstrated that the procedure improves performances on specialized data with a minimal loss in initial performances on generic domain for all languages pairs, compared to a naive standard approach (+10.0 BLEU score on specialized data, -0.01 to -0.5 BLEU on WMT and Tatoeba datasets on the other pairs with M2M100).