Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

TL;DR

This paper introduces Gate-DPO, a method that stabilizes preference optimization in language models by modulating gradients based on probability geometry, reducing training collapse and improving response quality.

Contribution

Gate-DPO is a novel gradient modulation technique that stabilizes preference optimization without altering the core objective, enhancing model alignment and training stability.

Findings

Gate-DPO reduces squeezing effects during training.

Smaller gated models outperform larger ungated models in response quality.

Mass-dynamics analysis shows healthier optimization with Gate-DPO.

Abstract

Preference optimization has become a central paradigm for aligning large language models with human feedback. Direct Preference Optimization (DPO) simplifies reinforcement learning from human feedback by directly optimizing pairwise preferences, removing the need for reward modeling and policy optimization. However, recent work shows that DPO exhibits a squeezing effect, where negative gradients applied to rejected responses concentrate probability mass on high-confidence predictions while suppressing alternative responses. This phenomenon arises even in simple softmax models and can lead to systematic probability collapse during training. We introduce Gradient-Gated Preference Optimization (Gate-DPO), a method that stabilizes training by modulating rejected gradients according to the model's probability geometry. When updates target extremely low-probability responses, the gate attenuates harmful gradients while preserving standard optimization behavior. Gate-DPO addresses this optimization pathology without modifying the underlying preference objective and is complementary to existing methods such as extended SFT, IPO, and Cal-DPO. Experiments across multiple architectures and preference datasets show that Gate-DPO consistently reduces squeezing and improves chosen-response likelihood. Mass-dynamics analysis further reveals healthier optimization behavior, with improved preferred responses and reduced suppression of the overall distribution. Notably, smaller gated models can exhibit stronger chosen-response improvements than larger ungated models, suggesting that controlling gradient dynamics, rather than scale alone, is key to stable and efficient alignment.