Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control

TL;DR

This paper introduces Anchored Learning, a distributional control framework for LLM fine-tuning that reduces catastrophic forgetting by stabilizing distributional updates, leading to improved performance and stability.

Contribution

It proposes a novel anchor-based method that interpolates between current and reference models, with theoretical guarantees and empirical validation on multiple benchmarks.

Findings

Significantly reduces performance degradation during fine-tuning.

Achieves near-optimal performance gains while maintaining stability.

Proven linear KL-divergence bound ensures stable distributional updates.

Abstract

Post-training large language models (LLMs) often suffers from catastrophic forgetting, where improvements on a target objective degrade previously acquired capabilities. Recent evidence suggests that this phenomenon is primarily driven by excessive distributional drift during optimization. Motivated by this perspective, we propose Anchored Learning, a simple framework that explicitly controls distributional updates during offline fine-tuning via a dynamically evolving moving anchor. Instead of matching a fixed reference distribution, the anchor interpolates between the current model and a frozen reference to construct an intermediate target that the model distills toward, transforming global fine-tuning into a sequence of local trust-region updates in distribution space. Theoretically, we prove this anchor-based update admits a linear KL-divergence upper bound per iteration, ensuring a stable transition between model distributions. Extensive experiments on iGSM, MedCalc, and IFEval show that Anchored Learning consistently lies on the Pareto frontier of gain-stability trade-offs, achieving near-optimal performance improvements while substantially reducing degradation compared to strong baselines. For example, while standard SFT suffers from over 53% performance degradation on iGSM and MedCalc, Anchored Learning slashes this drop to under 5% while maintaining near-optimal gains (e.g., 75.2% on iGSM).