Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models

TL;DR

This paper investigates the extent and nature of cross-domain transfer in pretrained models, revealing that BERT uniquely exhibits high transfer rates in classification tasks even with scrambled input, highlighting the roles of pretraining and word frequency.

Contribution

It systematically analyzes how much transfer occurs when models are deprived of word identity information, providing insights into the conditions and reasons for successful cross-domain transfer.

Findings

BERT shows high transfer rates in scrambled domains for classification tasks.

Transfer is limited in sequence labeling tasks and other models like LSTMs and GloVe.

Word frequency plays a significant role in transfer success.

Abstract

There is growing evidence that pretrained language models improve task-specific fine-tuning not just for the languages seen in pretraining, but also for new languages and even non-linguistic data. What is the nature of this surprising cross-domain transfer? We offer a partial answer via a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate baseline models, LSTMs using GloVe embeddings, and BERT. We find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, and to quantify the role of word frequency. These findings help explain where and why cross-domain transfer occurs, which can guide future studies and practical fine-tuning efforts.