Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority

TL;DR

This paper argues that effective supervised fine-tuning requires a focus on token priority to better align models with human utility, proposing a formal framework and analyzing recent advances through this lens.

Contribution

It introduces Token Priority as a formal mechanism for supervised fine-tuning, unifying recent breakthroughs into a coherent framework and highlighting future research directions.

Findings

Token Priority bridges the granularity gap in fine-tuning.

Recent advances fall into positive and signed priority regimes.

The framework clarifies progress and challenges in SFT.

Abstract

The transition from fitting empirical data to achieving true human utility is fundamentally constrained by a granularity mismatch, where fine-grained autoregressive generation is often supervised by coarse or uniform signals. This position paper advocates Token Priority as the essential bridge, formalizing Supervised Fine-Tuning (SFT) not as simple optimization but as a precise distribution reshaping process that aligns raw data with the ideal alignment manifold. We analyze recent breakthroughs through this unified lens, categorizing them into two distinct regimes: Positive Priority for noise filtration and Signed Priority for toxic modes unlearning. We revisit existing progress and limitations, identify key challenges, and suggest directions for future research.