The Trade-offs of Domain Adaptation for Neural Language Models

TL;DR

This paper analyzes the theoretical trade-offs in domain adaptation for neural language models, showing how data size and distribution affect performance and unifying various adaptation techniques within a common framework.

Contribution

It provides a theoretical analysis of domain adaptation, connecting it with machine learning theory, and unifies different data selection methods under a common framework.

Findings

Out-of-domain pre-training combined with in-domain fine-tuning improves generalization.

Performance depends on dataset sizes and distribution distances.

Various data selection techniques share underlying principles.

Abstract

This work connects language model adaptation with concepts of machine learning theory. We consider a training setup with a large out-of-domain set and a small in-domain set. We derive how the benefit of training a model on either set depends on the size of the sets and the distance between their underlying distributions. We analyze how out-of-domain pre-training before in-domain fine-tuning achieves better generalization than either solution independently. Finally, we present how adaptation techniques based on data selection, such as importance sampling, intelligent data selection and influence functions, can be presented in a common framework which highlights their similarity and also their subtle differences.