Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

TL;DR

This paper introduces DYPO, a unified framework that combines SFT and RL for large language models, effectively balancing bias and variance to improve reasoning and out-of-distribution performance.

Contribution

DYPO integrates group alignment loss, multi-teacher distillation, and dynamic gating to mitigate bias-variance conflict in LLM training, advancing beyond naive loss weighting strategies.

Findings

DYPO reduces reasoning benchmark errors by 4.8%.

DYPO improves out-of-distribution task performance by 13.3%.

Theoretical analysis confirms bias reduction and variance minimization.

Abstract

Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose \textbf{DYPO} (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a \textit{Group Alignment Loss (GAL)} that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a \textit{Multi-Teacher Distillation} mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a \textit{Dynamic Exploitation-Exploration Gating} mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8\% on complex reasoning benchmarks and 13.3\% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.