Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model

TL;DR

This paper introduces an unsupervised feature decomposition method to adapt pretrained cross-lingual language models to new domains, improving performance in cross-lingual and cross-domain tasks without labeled data.

Contribution

It proposes a novel mutual information-based unsupervised approach to extract domain-invariant and domain-specific features from pretrained models for better domain adaptation.

Findings

Significant performance improvements over state-of-the-art models in CLCD tasks

Effective unsupervised domain feature extraction from raw texts

Applicable to multiple languages and domains

Abstract

Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements over various cross-lingual and low-resource tasks. Through training on one hundred languages and terabytes of texts, cross-lingual language models have proven to be effective in leveraging high-resource languages to enhance low-resource language processing and outperform monolingual models. In this paper, we further investigate the cross-lingual and cross-domain (CLCD) setting when a pretrained cross-lingual language model needs to adapt to new domains. Specifically, we propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features and domain-invariant features from the entangled pretrained cross-lingual representations, given unlabeled raw texts in the source language. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts. Experimental results show that our proposed method achieves significant performance improvements over the state-of-the-art pretrained cross-lingual language model in the CLCD setting. The source code of this paper is publicly available at https://github.com/lijuntaopku/UFD.