Rotation-Preserving Supervised Fine-Tuning

TL;DR

RPSFT is a novel fine-tuning method that preserves pretrained singular subspaces to improve out-of-domain generalization and maintain representations.

Contribution

It introduces rotation-preserving regularization as an efficient proxy for Fisher-sensitive directions during supervised fine-tuning.

Findings

RPSFT improves in-domain/OOD trade-offs over standard SFT.

It better preserves pretrained representations.

Provides stronger initializations for downstream RL fine-tuning.

Abstract

Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that this degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-$k$ singular-vector block of each pretrained weight matrix, limiting unnecessary rotation while preserving task adaptation. Across model families and sizes trained on math reasoning data, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines, better preserves pretrained representations, and provides stronger initializations for downstream RL fine-tuning. Code is available at \href{https://github.com/jinhangzhan/RPSFT.git}{https://github.com/jinhangzhan/RPSFT}.