Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods

TL;DR

This paper investigates overadaptation in supervised fine-tuning of language models, revealing that ensembling pretrained and fine-tuned models not only preserves general knowledge but also enhances performance on the target domain, supported by theoretical analysis and experiments.

Contribution

It provides the first formal theoretical analysis of overadaptation in language models and demonstrates ensembling's effectiveness in balancing bias and variance during fine-tuning.

Findings

Ensembling pretrained and fine-tuned models improves performance.

Overadaptation phenomenon observed in language models.

Theoretical analysis supports empirical results.

Abstract

Supervised fine-tuning (SFT) on domain-specific data is the dominant approach for adapting foundation models to specialized tasks. However, it has been observed that SFT models tend to forget knowledge acquired during pretraining. In vision models, ensembling a pretrained model with its fine-tuned counterpart has been shown to mitigate this issue. In this work, we demonstrate that the same holds for language models, and, more strikingly, we observe an overadaptation phenomenon: the ensemble model not only retains general knowledge from the foundation model but also outperforms the fine-tuned model even on the fine-tuning domain itself. Despite the empirical success of ensembling, a theoretical understanding of its benefits remains underexplored. We develop a formal theoretical analysis of the overadaptation phenomenon. Ensembling mitigates this by balancing two primary sources of error: bias, caused by insufficient fine-tuning, and variance, introduced by overfitting to fine-tuning data. While regularization techniques aim to address this trade-off, we show that ensembling provides a more effective solution. We analyze this phenomenon in over-parameterized linear settings and demonstrate that interpolating between pretrained and fine-tuned weights significantly improves performance. These findings offer theoretical justification for the observed advantages of model ensembling, supported by empirical experiments consistent with our analysis.