Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?

TL;DR

This paper investigates why additional pretraining of large language models sometimes fails to improve performance, revealing that domain similarity and token informativeness influence adaptation success.

Contribution

It provides empirical insights into the conditions under which further training degrades LLM performance, highlighting the role of domain similarity and token informativeness.

Findings

Training on a domain can worsen test perplexity.

Performance degradation correlates with similarity to original pretraining data.

Token-level analysis shows a few uninformative tokens cause perplexity increase.

Abstract

In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLMs) are (i) even more overparameterized, (ii) trained on unlabeled text corpora curated from the Internet with minimal human intervention, and (iii) trained in an online fashion. These stark contrasts prevent researchers from transferring lessons learned on model generalization and adaptation in deep learning contexts to LLMs. To this end, our short paper introduces empirical observations that aim to shed light on further training of already pretrained language models. Specifically, we demonstrate that training a model on a text domain could degrade its perplexity on the test portion of the same domain. We observe with our subsequent analysis that the performance degradation is positively correlated with the similarity between the additional and the original pretraining dataset of the LLM. Our further token-level perplexity observations reveals that the perplexity degradation is due to a handful of tokens that are not informative about the domain. We hope these findings will guide us in determining when to adapt a model vs when to rely on its foundational capabilities.