Stabilizing Policy Optimization via Logits Convexity

TL;DR

This paper identifies the convexity of the supervised fine-tuning loss at the logits level as a key factor for stable reinforcement learning training, and introduces a new optimization method called Logits Convex Optimization (LCO) that enhances stability and performance.

Contribution

The paper reveals the importance of logits-level convexity for stable RL training and proposes LCO, a novel framework that improves stability and performance across various benchmarks.

Findings

LCO consistently improves training stability over traditional RL methods.

Logits convexity is crucial for favorable gradient directions during optimization.

Experiments show LCO outperforms PPO and other baselines on multiple benchmarks.

Abstract

While reinforcement learning (RL) has been central to the recent success of large language models (LLMs), RL optimization is notoriously unstable, especially when compared to supervised fine-tuning (SFT). In this work, we investigate the stability gap between SFT and RL from a gradient-based perspective, and show that the convexity of the SFT loss with respect to model logits plays a key role in enabling stable training. Our theoretical analysis demonstrates that this property induces favorable gradient directionality during optimization. In contrast, Proximal Policy Optimization (PPO), a widely adopted policy gradient algorithm utilizing a clipped surrogate objective, lacks this stabilizing property. Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework that aligns the learned policy with an optimal target derived from the original RL objective, thereby emulating the stabilizing effects of logits-level convexity. Extensive experiments across multiple model families show that our LCO framework consistently improves training stability and outperforms conventional RL methods on a broad range of benchmarks.