Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation

TL;DR

This paper explores how intermediate-task fine-tuning of pre-trained multilingual models can improve low-resource, domain-specific neural machine translation, especially for under-represented languages and limited data scenarios.

Contribution

It demonstrates that intermediate-task fine-tuning enhances domain adaptation in low-resource NMT and mitigates domain divergence effects for under-represented languages.

Findings

ITFT improves translation quality in low-resource settings

ITFT reduces domain divergence impact

Under-represented languages benefit from auxiliary data

Abstract

NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models. The problem gets aggravated when the data comes from different domains. In this paper, we show that intermediate-task fine-tuning (ITFT) of PMSS models is extremely beneficial for domain-specific NMT, especially when target domain data is limited/unavailable and the considered languages are missing or under-represented in the PMSS model. We quantify the domain-specific results variations using a domain-divergence test, and show that ITFT can mitigate the impact of domain divergence to some extent.