Learning While Staying Curious: Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

TL;DR

This paper introduces CurioSFT, an entropy-preserving supervised fine-tuning method that enhances exploration in large reasoning models, leading to improved performance in both fine-tuning and reinforcement learning stages.

Contribution

It proposes a novel entropy-preserving SFT approach with self-exploratory distillation and adaptive temperature selection to improve exploration and factual stability.

Findings

CurioSFT outperforms vanilla SFT by 2.5 and 2.9 points on in- and out-of-distribution tasks.

Enhanced exploration during SFT leads to a 5.0 point average improvement in RL stage.

The method effectively balances exploration and factual stability in large reasoning models.

Abstract

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.