Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

TL;DR

This paper explores how early data exposure during upstream training enhances the robustness of language models to retain capabilities after subsequent fine-tuning, emphasizing preventative training strategies.

Contribution

It demonstrates that early exposure during post-training improves capability retention after fine-tuning, supported by empirical and theoretical analysis across multiple models and tasks.

Findings

Early exposure improves the retention of upstream capabilities after fine-tuning.

Immediate post-training performance does not predict long-term retention reliably.

Replay and dropout methods complement early exposure in mitigating forgetting.

Abstract

How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.