Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives

TL;DR

This paper introduces a unified framework for supervised fine-tuning that adaptively balances learning from uncertain and confident predictions, improving model robustness and performance.

Contribution

It unifies token-level SFT objectives within a generalized deformed-log family and proposes DEFT, a parameter-free method that dynamically modulates trust in predictions based on entropy.

Findings

DEFT outperforms existing methods in balancing exploration and exploitation.

The universal gate-error gradient structure provides insights into model trust dynamics.

Experimental results show improved robustness and accuracy across tasks.

Abstract

Standard negative log-likelihood (NLL) for Supervised Fine-Tuning (SFT) applies uniform token-level weighting. This rigidity creates a two-fold failure mode: (i) overemphasizing low-probability targets can amplify gradients on noisy supervision and disrupt robust priors, and (ii) uniform weighting provides weak sharpening when the model is already confident. Existing methods fail to resolve the resulting plasticity--stability dilemma, often suppressing necessary learning signals alongside harmful ones. To address this issue, we unify token-level SFT objectives within a generalized deformed-log family and expose a universal gate $\times$ error gradient structure, where the gate controls how much the model trusts its current prediction. By employing the Cayley transform, we map the model's continuously evolving uncertainty onto a continuous focus trajectory, which enables seamless interpolation between scenarios involving uncertain novel concepts and those involving well-established knowledge. We then introduce Dynamic Entropy Fine-Tuning (DEFT), a parameter-free objective that modulates the trust gate using distribution concentration (R\'enyi-2 entropy) as a practical proxy for the model's predictive state. Extensive experiments and analyses demonstrate that DEFT achieves a better balance between exploration and exploitation, leading to improved overall performance.