Domain Specific Sub-network for Multi-Domain Neural Machine Translation

TL;DR

This paper introduces Domain-Specific Sub-networks (DoSS) for multi-domain neural machine translation, enabling efficient domain adaptation with fewer parameters and improved generalization to unseen domains.

Contribution

It proposes a pruning-based sub-network approach for domain-specific adaptation and a method for unique domain masks, enhancing performance and generalization in NMT.

Findings

Outperforms multi-domain baseline by 1.47 BLEU on German-English translation.

Achieves 1.52 BLEU improvement when adapting to a new domain.

Reduces parameters significantly compared to full fine-tuning.

Abstract

This paper presents Domain-Specific Sub-network (DoSS). It uses a set of masks obtained through pruning to define a sub-network for each domain and finetunes the sub-network parameters on domain data. This performs very closely and drastically reduces the number of parameters compared to finetuning the whole network on each domain. Also a method to make masks unique per domain is proposed and shown to greatly improve the generalization to unseen domains. In our experiments on German to English machine translation the proposed method outperforms the strong baseline of continue training on multi-domain (medical, tech and religion) data by 1.47 BLEU points. Also continue training DoSS on new domain (legal) outperforms the multi-domain (medical, tech, religion, legal) baseline by 1.52 BLEU points.