Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models

TL;DR

This paper introduces SAGE, a dynamic, stability-aware framework for preference-based alignment of reasoning models that improves training efficiency and stability by prioritizing informative and confident samples.

Contribution

SAGE presents a novel curriculum and scoring mechanism that adaptively selects training data based on model competence and stability, enhancing alignment reliability.

Findings

SAGE accelerates convergence compared to static methods.

SAGE outperforms baseline models on mathematical reasoning benchmarks.

Stability-aware data selection improves training efficiency.

Abstract

Preference-based alignment is pivotal for training large reasoning models; however, standard methods like Direct Preference Optimization (DPO) typically treat all preference pairs uniformly, overlooking the evolving utility of training instances. This static approach often leads to inefficient or unstable optimization, as it wastes computation on trivial pairs with negligible gradients and suffers from noise induced by samples near uncertain decision boundaries. Facing these challenges, we propose SAGE (Stability-Aware Gradient Efficiency), a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates. Concretely, SAGE integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence with a fine-grained, stability-aware scoring function that prioritizes informative, confident errors while filtering out unstable samples. Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines, highlighting the critical role of policy-aware, stability-conscious data selection in reasoning alignment.